Information processing device with flexible display, information processing method, and non-transitory computer readable medium

ABSTRACT

An information processing device includes a controller that controls an instruction to a control target associated with an image displayed at a changed position, according to a detection result of a change in each position on a deformable display. In a further modification of the invention, the controller may further change the image displayed at the changed position, according to the detection result.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation application of and claims thepriority benefit of a prior application Ser. No. 16/015,197 filed onJun. 22, 2018, now allowed. The prior application Ser. No. 16/015,197claims the priority benefit of Japan application serial no. 2017-204802,filed on Oct. 23, 2017. The entirety of each of the above-mentionedpatent applications is hereby incorporated by reference herein and madea part of this specification.

BACKGROUND Technical Field

The present invention relates to an information processing device, aninformation processing method, and a non-transitory computer readablemedium.

SUMMARY

According to an aspect of the invention, an information processingdevice includes a controller that controls an instruction to a controltarget associated with an image displayed at a changed position,according to a detection result of a change in each position on adeformable display.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments of the present invention will be described indetail based on the following figures, wherein:

FIG. 1 is a block diagram illustrating a configuration of an informationprocessing system according to a first exemplary embodiment of thepresent invention;

FIG. 2 is a block diagram illustrating a configuration of a terminaldevice;

FIG. 3 is a block diagram illustrating a configuration of a server;

FIG. 4 is a perspective diagram illustrating a flexible display;

FIG. 5 is a schematic diagram illustrating an external appearance of theimage forming device;

FIG. 6 is a diagram illustrating a device function management table;

FIG. 7 is a diagram illustrating a linkage function management table;

FIG. 8 is a schematic diagram illustrating a device and a terminaldevice;

FIG. 9 is a diagram illustrating a state where the flexible display isviewed from the bottom;

FIG. 10 is a schematic diagram illustrating a device and a terminaldevice;

FIG. 11 is a schematic diagram illustrating a device and a terminaldevice;

FIG. 12 is a schematic diagram illustrating a device and a terminaldevice;

FIG. 13 is a schematic diagram illustrating a device and a terminaldevice;

FIG. 14 is a schematic diagram illustrating a terminal device;

FIG. 15 is a schematic diagram illustrating a terminal device;

FIG. 16 is a schematic diagram illustrating a terminal device;

FIG. 17 is a schematic diagram illustrating a terminal device;

FIG. 18 is a diagram illustrating a linkage function management tableaccording to Example 4;

FIG. 19 is a schematic diagram illustrating a terminal device;

FIG. 20 is a diagram illustrating a device function management tableaccording to Example 5;

FIG. 21 is a schematic diagram illustrating a terminal device;

FIG. 22 is a diagram illustrating a device function management tableaccording to Example 6;

FIG. 23 is a diagram illustrating a linkage function management tableaccording to Example 6;

FIG. 24 is a schematic diagram illustrating a terminal device;

FIG. 25 is a schematic diagram illustrating a terminal device;

FIG. 26 is a sequence diagram illustrating an operation of theinformation processing system;

FIG. 27 is a schematic diagram illustrating a terminal device;

FIG. 28 is a diagram illustrating a linkage function management tableaccording to Example 9;

FIG. 29 is a schematic diagram illustrating a terminal device;

FIG. 30 is a diagram illustrating a linkage function management tableaccording to Example 10;

FIG. 31 is a schematic diagram illustrating a terminal device;

FIG. 32 is a schematic diagram illustrating a terminal device;

FIG. 33 is a diagram illustrating a linkage function management tableaccording to Example 12;

FIG. 34 is a schematic diagram illustrating a terminal device;

FIG. 35 is a schematic diagram illustrating a state where the flexibledisplay is viewed from the lateral side;

FIG. 36 is a schematic diagram illustrating a state where the flexibledisplay is viewed from the lateral side;

FIG. 37 is a schematic diagram illustrating a state where the flexibledisplay is viewed from the lateral side;

FIG. 38 is a schematic diagram illustrating a state where the flexibledisplay is viewed from the lateral side; and

FIG. 39 is a schematic diagram illustrating a state where the flexibledisplay is viewed from the lateral side.

DETAILED DESCRIPTION First Exemplary Embodiment

An information processing system according to a first exemplaryembodiment of the present invention will be described with reference toFIG. 1. FIG. 1 illustrates an example of the information processingsystem according to the first exemplary embodiment.

The information processing system according to the first exemplaryembodiment includes a terminal device 10, one or plural devices (e.g.,devices 12 and 14), and a server 16 as an example of an external device.In the example illustrated in FIG. 1, the terminal device 10, thedevices 12 and 14, and the server 16 have a function of communicatingwith each other via a communication path N such as a network. Of course,the terminal device 10, the devices 12 and 14, and the server 16 maycommunicate with other devices via different communication paths withoutusing the communication path N. In the example illustrated in FIG. 1,two devices (devices 12 and 14) are included in the informationprocessing system. However, one device may be included in theinformation processing system, or three or more devices may be includedin the information processing system. Further, plural terminal devices10 and plural servers 16 may be included in the information processingsystem. The server 16 may not be included in the information processingsystem.

The terminal device 10 is a device such as a personal computer (PC), atablet PC, a smartphone, a mobile phone, or the like, and has a flexibledisplay as a display device. Examples of the flexible display include anorganic electroluminescence display (flexible organic EL display), anelectronic paper display, a flexible liquid crystal display, and thelike. Flexible displays employing display modes other than these may beused. The flexible display is a display having a flexibly deformabledisplay portion, and is, for example, a display capable of being bent,folded, rolled, twisted, or stretched. The terminal device 10 may be awearable terminal (e.g., a wrist watch type terminal, a wrist band typeterminal, an eyeglass type terminal, a finger ring type terminal, acontact lens type terminal, an intracorporeal implantable terminal, ahearable terminal, or the like). In addition, the terminal device 10 hasa function of transmitting and receiving data to and from other devices.

The devices 12 and 14 are devices having functions, and include, forexample, an image forming device having an image forming function, a PC,a tablet PC, a smartphone, a mobile phone, a robot (a humanoid robot, ananimal robot other than humanoid, etc.), a projector, a display device(a liquid crystal display, etc.), a recording device, a reproducingdevice, an image capturing device (a camera, etc.), a refrigerator, arice cooker, a microwave oven, a coffee maker, a vacuum cleaner, awashing machine, an air conditioner, a lighting device, a clock, asurveillance camera, an automobile, a bicycle, an aircraft (e.g., anunmanned aerial vehicle (a so-called drone)), a game machine, andvarious types of sensing devices (e.g., a temperature sensor, a humiditysensor, a voltage sensor, a current sensor, etc.). The devices 12 and 14may be devices that output the output to a user (e.g., an image formingdevice or a PC), or may be devices that do not output an output to auser (e.g., a sensing device). Further, all of the devices in the pluraldevices that execute linkage functions described below may be devicesthat output an output to a user. Alternatively, some devices may bedevices that output an output to the user and other devices may bedevices that do not output any output to the user. Alternatively, all ofthe devices may be devices that do not output any output to the user.The concept of the devices 12 and 14 may include all kinds of devices.For example, an information device, a video device, an audio device, andother devices may also be included in the category of the devicesaccording to the first exemplary embodiment. In addition, the devices 12and 14 have a function of transmitting and receiving data to and fromother devices.

The server 16 is a device that manages data, manages user information,and manages the devices 12 and 14. In addition, the server 16 has afunction of transmitting and receiving data to and from other devices.

In the first exemplary embodiment, a change in each position on theflexible display is detected, and an instruction to a control targetassociated with an image displayed at the changed position is controlledaccording to the detected change. The control target is, for example, adevice, a function, a file, software, or the like. Among them, pluralobjects may be control targets. In the first exemplary embodiment, thecontrol target is a device.

Hereinafter, the configuration of the terminal device 10 will bedescribed in detail with reference to FIG. 2.

A communication unit 18 is a communication interface and has a functionof transmitting data to another device and a function of receiving datafrom another device. The communication unit 18 may be a communicationinterface having a wireless communication function or a communicationinterface having a wired communication function. For example, thecommunication unit 18 is compatible with one or plural types ofcommunication methods and may communicate with a communication partneraccording to a communication method suitable for the communicationpartner (i.e., a communication method with which the communicationpartner is compatible). The communication method is, for example,infrared communication, visible light communication, Wi-Fi (registeredtrademark) communication, proximity wireless communication (e.g., nearfield communication (NFC), etc.), and the like. Examples of theproximity wireless communication include Felica (registered trademark),Bluetooth (registered trademark), RFID (Radio Frequency Identifier), andthe like. Of course, other types of wireless communication may be usedas proximity wireless communication. The communication unit 18 mayswitch the communication method or the frequency band depending on thecommunication partner, or switch the communication method or thefrequency band depending on the surrounding environment.

A UI unit 20 is a user interface unit and includes a display unit and anoperation unit. The display unit is a flexible display. The operationunit is an input device such as a touch panel or a keyboard. Of course,it may be a user interface that serves as a display unit and anoperation unit (e.g., a touch-type display, a device for electronicallydisplaying a keyboard or the like on a display, etc.). Further, the UIunit 20 may include a voice collecting unit such as a microphone or avoice generating unit such as a speaker. In this case, information maybe input to the terminal device 10 by a voice input, or information maybe generated by voice.

The display unit of the UI unit 20 displays thereon, for example, animage captured by the camera, an image associated with a deviceidentified as a device to be used (e.g., a device used alone or a deviceto be linked), an image associated with a function, and the like. Theimage associated with the device may be an image (still image or movingimage) representing the device captured by the camera, or may be animage schematically representing the device (e.g., an icon). Forexample, the schematically indicated image data may be created by acontroller 28 of the terminal device 10, may be stored in advance in astorage unit 26 of the terminal device 10, may be stored in the server16 and provided from the server 16 to the terminal device 10, or may bestored in another device and provided from the other device to theterminal device 10. The image associated with a function is, forexample, an image such as an icon representing the function.

A camera 22 as an image capturing unit generates image data (e.g., stillimage data or moving image data) by capturing an object to be captured.The image is displayed on, for example, a display unit of the UI unit20. The image displayed on the display unit may be operated by the user.Image data captured by an external camera may be transmitted to theterminal device 10 via a communication path, and the image may bedisplayed on the display unit of the UI unit 20. Even in this case, theimage may be operated by the user.

A detection unit 24 has a function as a sensor for detecting a change ineach position on the flexible display. Examples of the sensor include apressure sensor, a temperature sensor, a humidity sensor, anacceleration sensor, a gyroscope sensor, and the like. The detectionunit 24 detects a bending change, a pressure change, a capacitancechange, an optical characteristic change, a temperature, a humidity, anacceleration, a direction, and the like for each portion of the flexibledisplay (e.g., for each pixel). The temperature and humidity may bedetected by sensors provided in devices other than the terminal device10 (e.g., devices 12, 14, etc.).

A storage unit 26 is a storage device such as a hard disk or memory(e.g., SSD etc.). The storage unit 26 stores, for example, devicefunction management information, linkage function managementinformation, various data, various programs (e.g., an operating system(OS), various application programs (application software), etc.),information indicating the address of each device (device addressinformation), information indicating the address of the server 16(server address information), information on the identified device,information on the identified device to be linked, information on solofunctions that the identified device has, information on the linkagefunctions, and the like. Of course, these may be stored in separatestorage devices or may be stored in one storage device.

Hereinafter, the device function management information and the linkagefunction management information will be described.

The device function management information is information for managing asolo function that each device has. The device function managementinformation includes, for example, information indicating an associationamong device identification information for identifying a device (deviceidentification information), solo function information indicating a solofunction that the device has, deformation operation informationindicating a deformation operation performed on the flexible display,and execution solo function information indicating a solo function(including processing and operation) executed by the device when thedeformation operation is performed among a solo function group that thedevice has.

The device identification information includes, for example, a deviceID, a device name, information indicating the type of the device, themodel number of the device, information for managing the device (e.g.,asset management number, etc.), information indicating the locationwhere the device is installed (position information of the device), adevice image associated with the device, address information of thedevice, and the like. The device image is, for example, an externalappearance image representing the device. The external appearance imagemay be an image representing the outside of the device (e.g., the casingof the device), an image representing a state where the casing is openedsuch that the inside thereof is seen from the outside (e.g., theinternal structure), or an image representing a state where the deviceis covered with a packaging sheet or the like. The device image may bean image generated by capturing the device (such as an imagerepresenting the outside of the device or an image representing theinside), or an image imitating the device (e.g., an icon).

The solo function information includes, for example, a function ID, afunction name, and the like. For example, when the device 12 is an imageforming device and the image forming device has a print function, a scanfunction, and a copy function, the device identification information ofthe image forming device has, as an example, solo function informationindicating the print function, function information indicating the scanfunction, and solo function information indicating the copy function,which are associated with each other.

The deformation operation is, for example, a bending operation, afolding operation, a winding operation, a twisting operation, astretching operation, or the like, performed on the flexible display. Inaddition, information indicating the direction of the deformationoperation (e.g., a bending direction, a folding direction, a windingdirection, a twisting direction, a stretching direction, etc.) may beincluded in the deformation operation information.

By referring to the device function management information, the solofunction that the device has is specified (identified), and theexecution solo function executed according to the deformation operationperformed on the flexible display is specified (identified).

The device managed by the device function management information is, forexample, a device (e.g., the device 12 or 14) included in theinformation processing system. Of course, devices not included in theinformation processing system may be managed by the device functionmanagement information. For example, the terminal device 10 may acquireinformation on a new device not included in the information processingsystem (information including device identification information, solofunction information, deformation operation information, and executionsolo function information) and newly register the information in thedevice function management information. The information on the devicemay be acquired, for example, by using the Internet or the like, or maybe input by a user, an administrator or the like. Further, the terminaldevice 10 may update the device function management information at anytiming, regularly, or at a timing designated by a user, an administratoror the like. Thus, in some cases, solo function information indicating asolo function that the device does not have before updating but thedevice has after updating may be registered in the device functionmanagement information. Similarly, in some cases, solo functioninformation indicating a solo function that the device has beforeupdating but the device does not have after updating may be deleted fromor registered as unusable information in the device function managementinformation. The information for updating may be acquired, for example,by using the Internet or the like, or may be input by a user, anadministrator or the like. Of course, the terminal device 10 itself maynot perform acquisition of information on the device and generation orupdate of the device function management information, but an externaldevice such as the server 16 may perform the acquisition of informationon the device and the generation or update of the device functionmanagement information. In this case, the device function managementinformation generated or updated by the external device is transmittedfrom the external device to the terminal device 10, and is stored in thestorage unit 26.

The linkage function management information is information for managinglinkage functions executed by linking plural functions to each other.When plural functions are linked to each other, one or plural linkagefunctions are executed. For example, the linkage functions may befunctions executable by linking plural functions that a single device(e.g., the device 12) has to each other, or solo functions executable bylinking plural functions that plural devices (e.g., the devices 12 and14) have to each other. Further, the terminal device 10 issuing theoperation instruction may also be used as a device to be linked, and thesolo function that the terminal device 10 has may also be used as a partof the linkage functions.

The linkage function may be a function executed without using a deviceas hardware. For example, the linkage function may be a functionexecuted by linking plural software to each other. Of course, thelinkage function may be a function executed by linking the function thatthe device as hardware has and the function implemented by software toeach other.

The linkage function management information is information indicating,for example, an association among a combination of solo functioninformation indicating respective solo functions used in the linkagefunction, linkage function information indicating the linkage function,deformation operation information indicating the deformation operationperformed on the flexible display, and execution linkage functioninformation indicating the linkage function (process or operation)executed when the deformation operation is performed.

The linkage function information includes, for example, a linkagefunction ID, a linkage function name, or the like. When a solo functionis updated, the linkage function management information is also updatedalong with the update. As a result, the linkage function by plural solofunctions that are unable to be linked to each other before updating maybecome usable after updating, and on the contrary to this, the linkagefunction that is usable before updating may become unusable afterupdating. The linkage function information indicating the linkagefunction that becomes usable after updating is registered in the linkagefunction management information, and the linkage function informationindicating the linkage function that became unusable after updating isdeleted from or registered as unusable information in the devicefunction management information. The update of the linkage functionmanagement information may be performed by the terminal device 10 or maybe performed by an external device such as the server 16. When theupdate of the linkage function management information is executed by anexternal device, the updated linkage function management information istransmitted from the external device to the terminal device 10, and isstored in the storage unit 26.

In the case of linking plural devices to each other, the linkagefunction management information is information for managing a linkagefunction executable using the plural devices, and is informationindicating an association among a combination of device identificationinformation for identifying the respective devices used in the linkagefunction, linkage function information indicating the linkage function,operation information indicating the deformation operation, andexecution linkage function information indicating a linkage functionexecuted when the deformation operation is performed, among a linkagefunction group executable using the plural devices. Further, in a caseof linking a device that has plural output functions (a device thatoutputs an output to a user and has plural output functions) and asensing device that does not have an output function (a device that doesnot output any output to a user), the linkage function managementinformation may include information indicating a combination of aspecific sensing device and a specific output function of the pluraloutput functions. Therefore, the output function used in combinationwith a specific sensing device is specified from among the plural outputfunctions.

The linkage function may be a function executable by linking plural solofunctions, which are different from each other, to each other, or bylinking identical solo functions to each other. The linkage function maybe a function that is unusable before the linkage. The function that isunusable before the linkage may be a solo function that is usable byusing an identical solo function among the functions that devices to belinked have or that is usable by combining different solo functions. Forexample, when a device having a print function (printer) and a devicehaving a scan function (scanner) are linked to each other, a copyfunction is implemented as a linkage function. That is, the copyfunction is implemented by linking the print function and the scanfunction to each other. In this case, the copy function as a linkagefunction and a combination of the print function and the scan functionare associated with each other. In the linkage function managementinformation, for example, linkage function information indicating thecopy function as a linkage function is associated with a combination ofdevice identification information for identifying a device having aprint function and device identification information for identifying adevice having a scan function.

Of course, the linkage function may be not only a function implementedby a combination of plural solo functions but also a functionimplemented by a combination of one or plural other linkage functionsand one or plural solo functions, or may be a function implemented by acombination of plural other linkage functions.

The concept of the linkage function may include a coalescing functionenabling a new function to be executed by linking plural solo functionsor plural devices to each other. For example, an extended displayfunction may be implemented as a coalescing function by combining pluraldisplays. As another example, a recording function may be implemented asa coalescing function by combining a television and a recorder. Therecording function may be a function of recording an image displayed onthe television. In addition, a capturing area extension function may beimplemented as a coalescing function by combining plural cameras. Theextension function is, for example, a function of capturing byconnecting the capturing areas of the respective cameras. In addition, atranslated call function (a function for translating a conversation viaa telephone) may be implemented as a coalescing function by combining atelephone with a translator or translation software. As described above,the concept of the linkage function includes a function that may beimplemented by linking the same types of devices or solo functions toeach other and a function that may be implemented by linking differenttypes of devices or solo functions to each other.

A controller 28 controls the operation of each unit of the terminaldevice 10. The controller 28 controls, for example, communication by thecommunication unit 18, display of information on the display unit of theUI unit 20, writing information to the storage unit 26, readinginformation from the storage unit 26, and the like. Further, thecontroller 28 includes a specifying unit 30.

The specifying unit 30 has a function of specifying solo functions thata device has or a linkage function executable using one or pluraldevices. Further, the specifying unit 30 has a function of, when adeformation operation is performed on the flexible display, specifyingan execution solo function corresponding to the deformation operation oran execution linkage function corresponding to the deformationoperation.

For example, the specifying unit 30 receives device identificationinformation for identifying the device and specifies solo functioninformation indicating a solo function associated with the deviceidentification information in the device function management informationstored in the storage unit 26. Thereby, the solo function that thedevice has is specified (identified). Further, when a deformationoperation is performed on the flexible display, the deformationoperation (e.g., a bending operation or the like) is detected by thedetection unit 24. The specifying unit 30 specifies execution solofunction information indicating an execution solo function associatedwith the deformation operation in the device function managementinformation. As a result, the solo function executed when thedeformation operation is performed on the flexible display is specified(identified).

The controller 28 controls notification (guidance) of information on thesolo function specified by the specifying unit 30 (e.g., solo functioninformation, execution solo function information, solo functionexplanation information, and the like). As the notification, thecontroller 28 may cause the display unit (flexible display) of the UIunit 20 to display the information on the solo function, or may use aspeaker or the like to issue information on the solo function as voiceinformation. Further, the controller 28 may control the execution of thesolo function. That is, the controller 28 may cause the device toexecute the solo function.

The process of specifying the solo function may be performed by theserver 16. In this case, the device identification information istransmitted from the terminal device 10 to the server 16, and the server16 specifies solo function information indicating a solo functionassociated with the device identification information. In the case wherethe specifying process is performed by the server 16, the devicefunction management information is stored in the server 16. Theinformation on the solo function may be transmitted from the server 16to the terminal device 10 and displayed on the display unit of the UIunit 20 or issued as voice information. Similarly, the process ofspecifying the execution solo function corresponding to the deformationoperation may be performed by the server 16. In this case, deformationoperation information is transmitted from the terminal device 10 to theserver 16, and the server 16 specifies execution solo functioninformation indicating an execution solo function associated with thedeformation operation information. The execution solo functioninformation may be transmitted from the server 16 to the terminal device10 and displayed on the display unit of the UI unit 20 or issued asvoice information.

Further, the specifying unit 30 receives the device identificationinformation for identifying respective devices to be linked andspecifies linkage function information indicating a linkage functionassociated with a combination of the respective device identificationinformation in the linkage function management information stored in thestorage unit 26. Therefore, the linkage function executable by linkingthe solo functions that the devices to be linked have is specified(identified). Further, when a deformation operation is performed on theflexible display, the deformation operation is detected by the detectionunit 24. The specifying unit 30 specifies execution linkage functioninformation indicating an execution linkage function associated with thedeformation operation in the linkage function management information.

The controller 28 controls notification (guidance) of information on thelinkage function specified by the specifying unit 30 (e.g., linkagefunction information, execution linkage function information, linkagefunction explanation information, and the like). The controller 28 maycause the display unit (flexible display) of the UI unit 20 to displaythe information on the linkage function, or may issue information on thelinkage function as voice information. Further, the controller 28 maycontrol the execution of the linkage function. That is, the controller28 may cause plural devices to be linked to execute the linkagefunction.

The specifying unit 30 may receive solo function information indicatingrespective solo functions used for the linkage function and specify thelinkage function information indicating the linkage function associatedwith a combination of the respective solo function information in thelinkage function management information. Therefore, the linkage functionexecutable by linking the respective solo functions to be linked isspecified (identified).

The process of specifying the linkage function may be performed by theserver 16. In this case, plural pieces of device identificationinformation are transmitted from the terminal device 10 to the server16, and the server 16 specifies linkage function information indicatinga linkage function associated with the plural pieces of deviceidentification information. In the case where the specifying process isperformed by the server 16, the linkage function management informationis stored in the server 16. The information on the linkage functions maybe transmitted from the server 16 to the terminal device 10 anddisplayed on the display unit of the UI unit 20 or issued as voiceinformation. Similarly, the process of specifying the execution linkagefunction corresponding to the deformation operation may be performed bythe server 16. In this case, deformation operation information istransmitted from the terminal device 10 to the server 16, and the server16 specifies execution linkage function information indicating anexecution linkage function associated with the deformation operationinformation. The execution linkage function information may betransmitted from the server 16 to the terminal device 10 and displayedon the display unit of the UI unit 20 or issued as voice information.

Functions usable by a user (a solo function or a linkage function) maybe managed for each user. The management may be performed by theterminal device 10 or may be performed by an external device such as theserver 16. The functions usable by a user include, for example, afunction provided free of charge to the user, a function provided to theuser for a fee and purchased by the user, and the like. Usable functioninformation indicating functions usable by the user (e.g., functionpurchase history information) may be created and managed for each user.When the management of the usable functions is performed by the terminaldevice 10, the usable function information is stored in the storage unit26. When the management is performed by an external device such as theserver 16, the usable function information is stored in the externaldevice. Of course, there are also functions that may be used for free,additional update functions, and functions that are specially managed byan administrator. Therefore, the usability of the function may not bedetermined based on the presence or absence of purchase. The process ofpurchasing the functions is performed, for example, by an externaldevice such as the server 16.

The usable function management information is information for managingthe functions usable by respective users. The usable function managementinformation is, for example, information indicating an associationbetween user identification information for identifying a user andfunction information indicating a function usable by the user (includingsolo function information and linkage function information). Asdescribed above, the function usable by the user is, for example, afunction provided free of charge to the user, a function purchased bythe user, and the like, which may be a single function or a linkagefunction. The user identification information is, for example, useraccount information such as user ID and name. The functions usable byeach user are specified (identified) by referring to the usable functionmanagement information. The usable function management information maybe updated, for example, each time when a function is provided to theuser (e.g., each time when a function is provided to the user free ofcharge or at a cost).

When the functions usable by a user are managed, the specifying unit 30may receive user identification information for identifying the user andspecify function information indicating respective functions associatedwith the user identification information in the usable functionmanagement information (e.g., the information stored in the terminaldevice 10 or server 16). As a result, functions usable by the user arespecified (identified). The controller 28 may cause the display unit ofthe UI unit 20 to display the information on the specified function, ormay issue the information as voice information. For example, thespecifying unit 30 receives the device identification information andthe user identification information, specifies solo function informationindicating a solo function associated with the device identificationinformation in the device function management information, and furtherspecifies function information indicating a function associated with theuser identification information (including a solo function and a linkagefunction) in the usable function management information. As a result,the solo function, that the device specified by the deviceidentification information has and that is usable by the user identifiedby the user identification information, is specified.

The functions usable by the user may be specified by an external devicesuch as the server 16. In this case, the user identification informationis transmitted from the terminal device 10 to the server 16, and theserver 16 specifies function information indicating respective functionsassociated with the user identification information. In the case wherethe specifying process is performed by the server 16, the usablefunction information is stored in the server 16. Information on eachfunction usable by the user may be transmitted from the server 16 to theterminal device 10 and displayed on the display unit of the UI unit 20or issued as voice information.

Hereinafter, the configuration of the server 16 will be described indetail with reference to FIG. 3. FIG. 3 illustrates the configuration ofthe server 16.

A communication unit 32 is a communication interface and has a functionof transmitting data to another device and a function of receiving datafrom another device. The communication unit 32 may be a communicationinterface having a wireless communication function or a communicationinterface having a wired communication function.

A storage unit 34 is a storage device such as a hard disk or memory(e.g., SSD etc.). The storage unit 34 stores, for example, various data,various programs, terminal address information of the terminal device10, device address information of each device, server addressinformation of the server 16, and the like. Of course, these may bestored in separate storage devices or may be stored in one storagedevice.

When the solo functions that the respective devices have and the linkagefunctions are managed by the server 16, the above-described devicefunction management information and linkage function managementinformation may be stored in the storage unit 34, and then, the devicefunction management information and the linkage function managementinformation may be updated in the server 16. In this case, when aprocess of specifying the device is performed by the terminal device 10,the device function management information and the linkage functionmanagement information (the updated information when the information isupdated) are transmitted from the server 16 to the terminal device 10.The transmission may be performed regularly or at a designated timing.Thus, the information stored in the terminal device 10 is updated. Ofcourse, the specifying unit 30 of the terminal device 10 may specifysolo functions or linkage functions by referring to the device functionmanagement information or the linkage function management informationstored in the server 16. Further, the process of specifying the functionmay be performed by the server 16. When the device function managementinformation is stored in the server 16, the device function managementinformation may not be stored in the terminal device 10. Similarly, whenthe linkage function management information is stored in the server 16,the linkage function management information may not be stored in theterminal device 10.

The controller 36 controls the operation of each unit of the server 16.The controller 36 controls, for example, communication by thecommunication unit 32, writing information to the storage unit 34,reading information from the storage unit 34, and the like. Further, thecontroller 36 may have the specifying unit 38 having the same functionas that of the specifying unit 30 of the terminal device 10. Similarlyto the specifying unit 30 of the terminal device 10, the specifying unit38 specifies a solo function and a linkage function. When the server 16performs the process of specifying the solo function and the linkagefunction, the terminal device 10 may not have the specifying unit 30.

The controller 36 may execute a process of purchasing a function andmanage the history of the purchase. For example, when a charged functionis purchased by a user, the controller 36 may apply a charging processto the user.

When usable function management information is created, the usablefunction management information may be stored in the server 16. In thiscase, the usable function management information may not be stored inthe terminal device 10. The controller 28 of the terminal device 10 maymanage the history of the purchase of functions by a user. In this case,the controller 36 of the server 16 may not have the management functionthereof.

As still another example, the device function management information andthe linkage function management information may be stored in the devicessuch as the devices 12 and 14, or the devices such as the devices 12 and14 may have the specifying unit 30. That is, the process by thespecifying unit 30 may be performed in the terminal device 10, thedevices such as the devices 12 and 14, the server 16, or another device.

Hereinafter, a flexible display provided in the terminal device 10 willbe described with reference to FIG. 4. FIG. 4 is a perspective diagramschematically illustrating an external appearance of a flexible display.In the example illustrated in FIG. 4, the terminal device 10 itselfconstitutes a flexible display, so that the terminal device 10 itself isdeformable. Of course, in the terminal device 10, the flexible displayand the other configurations may be separated from each other.

As an example, the terminal device 10 has a rectangular parallelepipedshape, and includes a front side display 40 provided on the frontsurface of the terminal device 10, a rear side display 42 (notillustrated in FIG. 4) provided on the rear surface thereof, lateralside displays 44 and 46 (the lateral side display 46 is not illustratedin FIG. 4) provided on the lateral surfaces thereof, an upper sidedisplay 48 provided on the upper surface thereof, and a lower sidedisplay 50 (not illustrated in FIG. 4) provided on the lower surfacethereof.

When an object to be displayed is a three-dimensional object, the frontside of the three-dimensional object may be displayed on the front sidedisplay 40, the rear side of the three-dimensional object may bedisplayed on the rear side display 42, the lateral sides of thethree-dimensional object may be displayed on the lateral side displays44 and 46, the upper side of the three-dimensional object may bedisplayed on the upper side display 48, and the lower side of thethree-dimensional object may be displayed on the lower side display 50.For example, an image 52 a representing the front side of thethree-dimensional object is displayed on the front side display 40, animage 52 b representing one lateral side of the three-dimensional objectis displayed on one lateral side display 44, and an image 52 crepresenting the upper side of the three-dimensional object is displayedon the upper side display 48. Corresponding images are also displayed onthe rear side display 42, the other lateral side display 46, and thelower side display 50, respectively.

When an object to be displayed is a three-dimensional object, thethree-dimensional object may be output by a 3D printer. Of course, evenwhen the object to be displayed is a three-dimensional object, the imageof the three-dimensional object may be converted into a two-dimensionalplanar image (for example, converted by projection) and output as aplanar image. Further, the object to be displayed may be printed foreach individual side, or the object to be displayed may be printedbefore and after the change of the object to be displayed.

The flexible display holds image information for each portion (e.g., foreach pixel). For example, the flexible display holds image informationfor each unit portion (e.g., pixel) in a three-dimensional space (e.g.,a space defined by X axis, Y axis, and Z axis orthogonal to each other).As another example, the flexible display may manage the imageinformation for each two-dimensional plane (two-dimensional layer).

As described above, the detection unit 24 detects a change in eachposition of the flexible display (the terminal device 10 itself in theexample illustrated in FIG. 4). In the example illustrated in FIG. 4,the detection unit 24 detects changes in the front surface, the rearsurface, both lateral surface, the upper surface, and the lower surface.

The flexible display illustrated in FIG. 4 is merely an example. Thedisplay units may not be provided on the rear surface, the lateralsurfaces, the upper surface, and the lower surface. Alternatively, thedisplay units may be provided on the front surface and the rear surface.Further, the flexible display may be a display having a wholly orpartially curved display unit (a display unit having a curved surface),a winding type display, or a slide type display. Even a display otherthan these displays may be included in the category of the flexibledisplay according to the first exemplary embodiment as long as it is adisplay capable of changing its shape.

In the first exemplary embodiment, as an example, the deviceidentification information is acquired by using an augmented reality(AR) technology so that the device is identified. For example, by usingthe AR technology, the device identification information of the deviceused alone is acquired, so that the device is identified, and the deviceidentification information of the device to be linked is acquired, sothat the device to be linked is identified. As the AR technology, any ARtechnology of the related art is used. Examples of the AR technologyinclude an image AR technology using image analysis, a position ARtechnology using position information, and the like. Examples of theimage AR technology include a marker AR technology using a marker suchas a two-dimensional barcode, a markerless AR technology using an imagerecognition technology, and the like. Of course, the deviceidentification information may be acquired without using the ARtechnology, and then the device may be identified. For example, when thedevice is connected to the network, the device may be identified basedon the IP address, or the device ID may be read to identify the device.Further, in a case of using a device or a terminal device having variouswireless communication functions such as infrared communication, visiblelight communication, Wi-Fi, Bluetooth, etc., the ID of the device may beacquired using the wireless communication function, so that the deviceis identified.

Hereinafter, the process of acquiring the device identificationinformation will be described in detail with reference to FIG. 5. FIG. 5schematically illustrates the external appearance of a multifunctiondevice 54 as an exemplary device. The multifunction device 54 is, forexample, a device having at least one function of a print function, ascan function, a copy function, or a facsimile function. Here, a processfor acquiring device identification information of the multifunctiondevice 54 by applying the marker AR technology will be described. Amarker 56 such as a two-dimensional barcode or the like is provided onthe casing of the multifunction device 54. The marker 56 is informationin which the device identification information of the multifunctiondevice 54 is encoded. When a user captures the marker 56 with the camera22 of the terminal device 10, image data representing the marker 56 isgenerated. The specifying unit 30 of the terminal device 10 extracts thedevice identification information by applying a decoding process to themarker image represented in the image data. Thereby, the multifunctiondevice 54 is identified. Next, the specifying unit 30 specifies solofunction information indicating a solo function associated with theextracted device identification information in the device functionmanagement information. Thereby, the solo function that themultifunction device 54 has is specified (identified).

The specifying unit 38 of the server 16 may extract the deviceidentification information by applying the decoding process to the imagedata representing the marker 56. In this case, image data is transmittedfrom the terminal device 10 to the server 16, and the decoding processis applied to the image data in the server 16. Further, in the server16, the solo function associated with the device identificationinformation may be specified.

Solo function information indicating the solo function that themultifunction device 54 has may be coded and included in the marker 56.In this case, when the decoding process is applied to the image datarepresenting the marker 56, the device identification information of themultifunction device 54 is extracted, and the solo function informationindicating the solo function that the multifunction device 54 has isextracted as well. As a result, the multifunction device 54 is specified(identified), and the solo function that the multifunction device 54 hasis specified (identified). The decoding process may be performed by theterminal device 10 or may be performed by the server 16. Further,deformation operation information and execution solo functioninformation may be coded and included in the marker 56.

In a case of acquiring the device identification information by applyingthe markerless AR technology, for example, the user captures theentirety or a part of the external appearance of the device (e.g., themultifunction device 54) with the camera 22 of the terminal device 10.Of course, it is useful to acquire information for identifying a devicesuch as the name (e.g., product name), a model number, or an assetmanagement number of the device, from the external appearance. Externalappearance image data representing the entirety or a part of theexternal appearance of the device is generated by image capturing. Thespecifying unit 30 of the terminal device 10 identifies the device basedon the external appearance image data. For example, the storage unit 26of the terminal device 10 stores, for each device, external appearanceimage association information indicating an association between theexternal appearance image data representing the entirety or a part ofthe external appearance of the device and the device identificationinformation of the device. For example, the specifying unit 30 comparesthe external appearance image data with each external appearance imagedata included in the external appearance image association information,and specifies the device identification information of the device basedon the comparison result. For example, the specifying unit 30 extractsfeatures of the external appearance of the device from the appearanceimage data, specifies the external appearance image data representingthe same or similar features as or to the features of the externalappearance in the external appearance image data group included in theexternal appearance image association information, and specifies thedevice identification information associated with the externalappearance image data. Thereby, the device is identified. As anotherexample, when the name (e.g., product name) or model number of thedevice is captured and external appearance image data representing thename or model number is generated, the device may be identified based onthe name or model number indicated in the external appearance imagedata. The specifying unit 30 specifies solo function informationindicating a solo function associated with the specified deviceidentification information in the device function managementinformation. Thereby, the solo function that the device has isspecified.

The specifying unit 38 of the server 16 may compare the externalappearance image data representing all or a part of the externalappearance of the device with each external appearance image dataincluded in the external appearance image association information, andspecify the device identification information of the device based on thecomparison result.

When acquiring the device identification information by applying theposition AR technology, position information indicating a position wherethe device is installed is acquired by using, for example, the globalpositioning system (GPS) function. For example, each device has the GPSfunction and acquires device position information indicating theposition of the device itself. The terminal device 10 outputsinformation indicating a device position information acquisition requestto the device to be used, and receives device position information ofthe device from the device as a response to the acquisition request. Thespecifying unit 30 of the terminal device 10 identifies the device basedon the device position information. For example, the storage unit 26 ofthe terminal device 10 stores, for each device, position associationinformation indicating an association between the device positioninformation indicating the position where the device is installed andthe device identification information of the device. The specifying unit30 specifies the device identification information associated with thedevice position information in the position association information.Thereby, the device is specified (identified). The specifying unit 30specifies solo function information indicating a solo functionassociated with the specified device identification information in thedevice function management information. Thereby, the solo function thatthe device has is specified (identified).

The specifying unit 38 of the server 16 may specify the deviceidentification information associated with the position information ofthe device in the position association information.

Devices may be identified using plural identification technologies. Forexample, the devices may be identified using plural technologiesselected from the marker AR technology, the markerless AR technology,and the position AR technology. Further, when a device is not identifiedby a certain identification technology, the device may be identifiedusing another identification technology. For example, when a device isnot identified by the marker AR technology or the markerless ARtechnology, the device may be identified using the position ARtechnology.

Hereinafter, the information processing system according to the firstexemplary embodiment will be described in more detail.

The device function management information will be described in detailwith reference to FIG. 6. FIG. 6 illustrates an exemplary devicefunction management table as the device function management information.In the device function management table, as an example, the device ID,the information indicating the device name (e.g., the type of thedevice), the information indicating the solo function that the devicehas (solo function information), the deformation operation informationindicating the deformation operation performed on the flexible display,the execution solo function information which is a solo function to beexecuted by the device when the deformation operation is performed,among the solo function group that the device has, and the image ID areassociated with each other. The device ID and the device name areexamples of the device identification information. The image ID is anexample of image identification information for identifying a deviceimage associated with the device. The device function management tablemay not include the image ID.

For example, a device whose device ID is “B” is a multifunction device,and has solo functions such as a print function and a scan function.Further, the multifunction device is associated with an image ID foridentifying a device image associated with the multifunction device. Thedata of the device image associated with the device may be stored in,for example, the terminal device 10, the server 16, or another device.

For example, the device ID for identifying the device (e.g., themultifunction device (B)) is acquired by applying the AR technology. Thespecifying unit 30 of the terminal device 10 specifies the device name,the solo function, and the image ID associated with the device ID byreferring to the device function management table. Thereby, themultifunction device (B) is identified. Information indicating thedevice name or the device image may be displayed on the display unit ofthe UI unit 20. Of course, the image itself captured by the camera 22may be displayed on the display unit of the UI unit 20. Further, whenthe device image associated with the multifunction device (B) (e.g., animage captured by the camera 22 or an image schematically representingthe multifunction device (B)) is designated by the user in the UI unit20, information on a solo function that the multifunction device (B) has(e.g., solo function information, explanation information on the solofunction, etc.) may be displayed on the display unit of the UI unit 20.Further, when a specific deformation operation is performed by the useron the flexible display in a state where the multifunction device (B) isidentified and displayed on the flexible display, the specifying unit 30specifies the execution solo function corresponding to the deformationoperation. The execution solo function may be executed by themultifunction device (B), or information indicating the execution solofunction may be displayed on the flexible display.

Each device is associated with one or plural deformation operations, andeach deformation operation is associated with the same or differentexecution solo functions. For example, it is assumed that themultifunction device (B) is associated with plural deformationoperations (e.g., deformation operations b1, b2, . . . ), thedeformation operation b1 is associated with a print function, and thedeformation operation b2 is associated with a scan function. Further,when the deformation operation b1 performed on the flexible display isdetected by the detection unit 24, the specifying unit 30 specifies theprint function as an execution solo function associated with thedeformation operation b1. In this case, the controller 28 may cause thedisplay unit of the UI unit 20 to display information indicating theprint function, may output the information as voice information, or maycause the multifunction device (B) to execute printing by giving a printinstruction to the multifunction device (B).

Hereinafter, the linkage function management information will bedescribed in detail with reference to FIG. 7. FIG. 7 illustrates anexemplary linkage function management table as the linkage functionmanagement information. In the linkage function management table, as anexample, a combination of device IDs, information indicating the devicenames (e.g., the type of the devices), information indicating singlefunctions that the devices have (single function information),deformation operation information indicating the deformation operationperformed on the flexible display, and execution linkage functioninformation indicating a linkage function to be executed when thedeformation operation is performed, among a linkage function groupexecutable using plural devices to be linked are associated with eachother.

For example, the device whose device ID is “A” is a PC (personalcomputer), and the device whose device ID is “B” is a multifunctiondevice. When the PC (A) and the multifunction device (B) are linked toeach other, for example, a “scan transfer function” and a “printfunction” are implemented as linkage functions. The “scan transferfunction” is a function of transferring image data generated by scanningby the multifunction device (B) to the PC (A). The “print function” is afunction of transmitting data (e.g., image data and document data)stored in the PC (A) to the multifunction device (B) and printing itwith the multifunction device (B). A linkage function executable usingthree or more devices may be specified. In this case, a combination ofthree or more devices and a linkage function are associated with eachother, and the association is registered in the linkage functionmanagement table.

A combination of the plural devices is associated with one or moredeformation operations, and each deformation operation is associatedwith the same or different execution linkage functions. For example, acombination of the PC (A) and the multifunction devices (B) isassociated with plural deformation operations, and each deformationoperation is associated with the same or different execution linkagefunctions. When a certain deformation operation is detected by thedetection unit 24, the specifying unit 30 specifies an execution linkagefunction associated with the deformation operation. In this case, thecontroller 28 may cause the display unit of the UI unit 20 to displayinformation indicating the execution linkage function, may output theinformation as voice information, or may cause the PC (A) and themultifunction device (B) to execute the execution linkage function bygiving an execution instruction thereof to the multifunction device (B).

Hereinafter, specific examples of the first exemplary embodiment will bedescribed in detail.

Example 1

Example 1 will be described. In FIG. 8, the robot 58 and the terminaldevice 10 are illustrated. The robot 58 is an exemplary device. Therobot 58 and the terminal device 10 are devices existing in a realspace. The robot 58 is identified by the specifying unit 30, using theabove-mentioned AR technology or other identification technologies. Inthe example illustrated in FIG. 8, the robot 58 has a face portionfacing forward and faces the user (terminal device 10).

A device image 60 is displayed on the front side display 40 (an exampleof the display unit of the UI unit 20) of the terminal device 10. Thedevice image 60 is an image associated with the robot 58. The deviceimage 60 may be an image generated by capturing the robot 58 (an imagehaving a size at the time of image capturing, or an enlarged or reducedimage) or an image schematically representing the robot 58 (e.g., anicon). The device image 60 is an image representing the externalappearance state of the robot 58. In the example illustrated in FIG. 8,the image represents the state of the external appearance of the robot58 with the face portion facing forward. The user may operate the UIunit 20 to change the display position and display size of the deviceimage 60.

When a schematic image is used, the specifying unit 30 specifies aschematic image associated with the identified device by referring tothe device function management table illustrated in FIG. 6. Theschematic image is displayed as a device image. The data of theschematic image may be stored in the terminal device 10 or may be storedin an external device such as the server 16.

The image data generated by capturing the device is image data thatreflects the external appearance of the current device itself (e.g.,scratches, memo, or external appearance reflecting a seal affixed on thedevice, etc.), so that the user may see the difference visually fromother similar devices.

Further, when a device is identified, information indicating the name ofthe device (corresponding to an example of device identificationinformation) may be displayed. For example, the character string “robot”may be displayed together with the device image 60. The device image 60may not be displayed, and information indicating the name of the robot58 may be displayed.

In addition, the information on the devices identified in the past maybe stored in the storage unit 26 of the terminal device 10. Thereafter,even though the identifying process is not performed again on thedevice, the device image associated with the device may be displayed. Ofcourse, the information on the identified device may be deleted from thestorage unit 26 by the operation of the user. In this case, the deviceimage associated with the device is also deleted. As another example,after a solo function or a linkage function is executed using theidentified device, or after a predetermined time elapses from the timepoint when the device is identified, information on the identifieddevice may be deleted from the storage unit 26.

When the user performs a deformation operation on the flexible display(the terminal device 10 itself in the example illustrated in FIG. 8)while the device image 60 is displayed, the detection unit 24 of theterminal device 10 detects the deformation operation, and the specifyingunit 30 specifies the execution solo function associated with thedeformation operation in the device function management tableillustrated in FIG. 6. The controller 28 causes the robot 58 to executethe execution solo function. When the controller displays informationindicating the execution solo function on the flexible display and theuser gives an execution instruction, the controller 28 may cause therobot 58 to execute the execution solo function. Further, the controller28 changes the device image 60 according to the deformation operation.Hereinafter, the process will be described in detail.

The detection unit 24 of the terminal device 10 detects the change(deformation) of the flexible display and specifies the position wherethe change occurs. Next, the controller 28 specifies the content (e.g.,the device image 60) displayed at the position where the change occurs,and changes the substance of the content. When the device image is notdisplayed and information indicating the content (e.g., a characterstring indicating the name of the robot 58) is displayed, the controller28 may specify the information indicating the content displayed at theposition where the change occurs (e.g., a character string). Forexample, the controller 28 may add a color or image to the content,delete a part or all of the content, or replace the content with othercontent (another image). When a new deformation operation is notdetected after the content is changed, the controller 28 may maintainthe displayed content after the change, or may return the displayedcontent to the content before the change after a predetermined timeelapses. Further, the specifying unit 30 specifies the device (e.g., therobot 58) associated with the content and specifies the execution solofunction associated with the deformation operation. The controller 28causes the device associated with the content to execute the executionsolo function. The order of the process of changing the content and theprocess of specifying the device associated with the content may bereversed from the above order, or the processes may be performedsimultaneously.

Hereinafter, a specific example will be described. It is assumed that auser bends the whole of a left side portion 10 a of the flexible display(terminal device 10 itself) toward a right side portion 10 b of theflexible display. That is, it is assumed that the user bends theflexible display in the direction of an arrow 62 illustrated in FIG. 8.FIG. 9 illustrates a state when the terminal device 10 is viewed fromthe lower side (the lower side display 50 side) of the terminal device10. The left side portion 10 a of the flexible display is bent in thedirection of the arrow 62 (the direction toward the right side portion10 b).

The detection unit 24 detects the change (deformation) of the flexibledisplay caused by the above-described bending operation, and specifiesthe position where the change occurs. The controller 28 specifies thedevice image displayed at that position. When a change occurs at theposition where the device image 60 is displayed due to theabove-described bending operation, the controller 28 identifies thedevice image 60 as the image to be changed. Further, the specifying unit30 identifies the robot 58 associated with the device image 60 as acontrol target, and refers to the device function management tableillustrated in FIG. 6 to specify the execution solo function associatedwith the bending operation (one mode of the deformation operation) amongthe solo function group that the robot 58 has. For example, it isassumed that the execution solo function is a function (an operation ora process) of directing the face portion of the robot 58 in thedirection in which the flexible display is bent.

The controller 28 transmits to the robot 58 control information forcausing the robot 58 to execute the execution solo function specified asdescribed above. For example, the controller 28 acquires addressinformation of the robot 58, establishes communication with the robot 58using the address information, and transmits the control information tothe robot 58. According to the control information, the robot 58 movesits own head (rotates the head about the neck) as illustrated in FIG.10, to direct the face portion in the same direction of an arrow 64 (theright direction as viewed from the user) as the direction in which theflexible display is bent (the direction of the arrow 62). Further, thecontroller 28 directs the head image in the device image 60 in thedirection of an arrow 66 (the right direction as viewed from the user)which is the same as the direction of the arrow 62. As a result, thedevice image 60 after the change represents the actual state of therobot 58 (the state where the face portion faces rightward when viewedfrom the user).

The controller 28 may control notification of the specified executionsolo function and may cause the robot 58 to execute the execution solofunction when the execution instruction is given by the user. Forexample, as the notification of the execution solo function, thecontroller 28 may cause the flexible display to display informationindicating the execution solo function, or may output informationindicating the execution solo function as voice information.

According to Example 1, when the user deforms the flexible display, afunction (an operation or a process) corresponding to the deformation isexecuted by the device. Thereby, with a simple operation, the functionis executed by the device. Further, the result of execution of thefunction is reflected in the device image.

Further, the device may operate in conjunction with the speed of thedeformation operation. In the above example, as the bending speed of theflexible display increases, the rotational speed of the head of therobot 58 increases.

The controller 28 may change the execution solo function according tothe mode of the deformation operation (the mode of change of theflexible display detected by the detection unit 24). For example, theoperation of folding the flexible display and the operation of twistingthe flexible display are associated with different execution solofunctions, respectively, and the specifying unit 30 specifies theexecution solo functions associated with the detected deformationoperation. When different execution solo functions are associated withthe respective modes of the deformation operation, various solofunctions may be executed with a simple operation. Of course, the sameexecution solo function may be associated with the respectivedeformation operations. Such an association may be changed by a user, anadministrator, or the like.

When the device image is changed by the user's deformation operation andwhen the device associated with the device image is not able to becontrolled, the controller 28 of the terminal device 10 returns thechanged device image to the original device image and displays theoriginal device image. The reason why the control is impossible is notparticularly limited, but for example, control is impossible due to acommunication error or the like. For example, as illustrated in FIG. 8,when the user bends the flexible display in the direction of the arrow62 as illustrated in FIG. 8, the controller 28 directs the head image inthe device image 60 in the direction of the arrow 66 as illustrated inFIG. 11. When the robot 58 is unable to be controlled by the terminaldevice 10, the robot 58 does not execute the operation according to thedeformation operation. For example, as illustrated in FIG. 11, the robot58 faces the front without directing the face portion to the directionof the deformation operation. In this case, as illustrated in FIG. 12,the controller 28 returns the changed device image 60 to the originaldevice image 60 so as to cause a flexible display to display the deviceimage 60 in a state where the head image faces forward. In this manner,the device image 60 representing the state of the current externalappearance of the robot 58 is displayed.

As another example, the controller 28 causes the robot 58 to execute theexecution solo function corresponding to the deformation operation, andupon receiving information indicating that the execution is completedfrom the robot 58 (receiving feedback from the robot 58), the controller28 may change the device image 60. In the above example, when the robot58 directs the face portion in the direction of the arrow 64 (see FIG.10), information indicating to that effect is transmitted to theterminal device 10. Upon receiving the information, the controller 28directs the head image in the device image 60 in the direction of thearrow 66 (see FIG. 10).

Example 2

Example 2 will be described. In FIG. 13, the robot 58, the multifunctiondevice 68, and the terminal device 10 are illustrated. The robot 58 andthe multifunction device 68 are exemplary devices. The robot 58, themultifunction device 68, and the terminal device 10 are devices existingin a real space. The robot 58 and the multifunction device 68 areidentified by the specifying unit 30, using the above-mentioned ARtechnology or other identification technologies.

Device images 60 and 70 are displayed on the front side display 40 ofthe terminal device 10. The device image 70 is an image associated withthe multifunction device 68. The device image 70 may be an imagegenerated by capturing the multifunction device 68, or an imageschematically representing the multifunction device 68 (e.g., an icon).The user may operate the UI unit 20 to change the display positions anddisplay sizes of the device images 60 and 70. Character stringsindicating the names of the robot 58 and the multifunction device 68 maybe displayed along with the device images 60 and 70 or in place of thedevice images 60 and 70.

In the case where the user performs a deformation operation on theflexible display (the terminal device 10 itself in the exampleillustrated in FIG. 13) while displaying plural device images (thedevice images 60 and 70 in the example illustrated in FIG. 13), thespecifying unit 30 of the terminal device 10 specifies an executionlinkage function associated with the deformation operation among thelinkage function group executable using the robot 58 and themultifunction device 68, in the linkage function management tableillustrated in FIG. 7. The controller 28 causes the robot 58 and themultifunction device 68 to execute the execution linkage function. Whenthe controller displays information indicating the execution linkagefunction on the flexible display and the user gives an executioninstruction, the controller 28 may cause the robot 58 and themultifunction device 68 to execute the execution linkage function.Further, the controller 28 changes the device images 60 and 70 accordingto the deformation operation.

For example, it is assumed that the user bends the whole of the leftside portion 10 a of the flexible display (terminal device 10 itself)toward the right side portion 10 b of the flexible display. That is, itis assumed that the user bends the flexible display in the direction ofan arrow 72.

The detection unit 24 detects the change (deformation) of the flexibledisplay caused by the above-described bending operation, and specifiesthe position where the change occurs. The controller 28 specifies thedevice image displayed at that position. When changes occur at thepositions where the device images 60 and 70 are displayed due to theabove-described bending operation, the specifying unit 30 identifies therobot 58 associated with the device image 60 and the multifunctiondevice 68 associated with the device image 70 as control targets(objects to be linked) and refers to the linkage function managementtable illustrated in FIG. 7 to thereby specify the execution linkagefunction associated with the bending operation among the linkagefunction group executable using the robot 58 and the multifunctiondevice 68. For example, it is assumed that the execution linkagefunction is a function (an operation or a process) of having the robotbring paper discharged to the multifunction device.

As illustrated in FIG. 14, the controller 28 causes the front sidedisplay 40 to display information indicating the execution linkagefunction specified as described above. The controller 28 may outputinformation indicating the execution linkage function as voiceinformation. When the user gives an execution instruction of theexecution linkage function, the controller 28 acquires addressinformation of each of the robot 58 and the multifunction device 68,establishes communication with the devices to be linked (the robot 58and the multifunction device 68) using the respective addressinformation, and transmits control information for causing the robot 58and the multifunction device 68 to execute the execution linkagefunction to the robot 58 and the multifunction device 68. The robot 58and the multifunction device 68 execute the execution linkage functionaccording to the control information. In the case where theabove-described bending operation is performed, the controller 28 maycause the robot 58 and the multifunction device 68 to execute theexecution linkage function even when the execution instruction is notgiven by the user.

According to Example 2, when the user deforms the flexible display, thelinkage function according to the deformation is notified or executed.Thereby, the linkage function is notified or executed by a simpleoperation.

The controller 28 may change the linkage function according to the modeof the deformation operation (the mode of change of the flexible displaydetected by the detection unit 24). For example, the operation offolding the flexible display and the operation of twisting the flexibledisplay are associated with different execution linkage functions, andthe specifying unit 30 specifies the execution linkage functionsassociated with the detected deformation operation. When differentexecution linkage functions are associated with the respective modes ofthe deformation operation, various linkage functions may be executedwith a simple operation. Of course, the same execution linkage functionmay be associated with the respective deformation operations. Such anassociation may be changed by a user, an administrator, or the like.

For example, the controller 28 may change the linkage function accordingto the mode of the bending operation (the mode of bending change of theflexible display detected by the detection unit 24). For example, asillustrated in FIG. 15, it is assumed that the user bends the whole ofthe right side portion 10 b of the flexible display toward the left sideportion 10 a of the flexible display. That is, it is assumed that theuser bends the flexible display in the direction of an arrow 74 (thedirection opposite to the arrow 72 illustrated in FIG. 13). In thiscase, the specifying unit 30 specifies an execution linkage functionassociated with the bending operation. For example, the operation ofbending the flexible display in the direction of the arrow 74 isassociated with an execution linkage function different from that of theoperation of bending the flexible display in the direction of the arrow72, and the specifying unit 30 specifies the execution linkage function.Thereby, various linkage functions may be executed by changing thedirection in which the bending operation is performed. The controller 28may control the notification of the information indicating the executionlinkage function, or may cause the robot 58 and the multifunction device68 to execute the execution linkage function.

When three or more devices are identified as control targets (objects tobe linked) by the deformation operation, an execution linkage functionassociated with the deformation operation is selected from a linkagefunction group executable using the three or more devices, andnotification or execution of the execution linkage function iscontrolled.

Example 3

Example 3 will be described. FIG. 16 illustrates the terminal device 10.Similarly to Example 2, the robot 58 and the multifunction device 68 areidentified, and the device image 60 associated with the robot 58 and thedevice image 70 associated with the multifunction device 68 aredisplayed on the front side display 40.

When the user bends an end portion (corner portion) 76 of the flexibledisplay (the terminal device 10 itself) toward the front side display 40(in the direction indicated by an arrow 78), the controller 28 specifiesthe device image 60 displayed on an extension line 80 in the directionof the bending operation (the direction indicated by the arrow 78) asthe device image associated with the control target. The extension line80 has a width corresponding to one or plural pixels. The width of theextension line 80 may be set to any value by a user, an administrator,or the like. Further, the controller 28 may specify the device imagedisplayed on the extension line in the direction of the force applied tothe flexible display by the bending operation as the device imageassociated with the control target. In the example illustrated in FIG.16, since the device image 60 is displayed on the extension line 80, thespecifying unit 30 identifies the robot 58 associated with the deviceimage 60 as a control target. Next, by referring to the device functionmanagement table illustrated in FIG. 6, the specifying unit 30 specifiesthe execution solo function associated with the above-described bendingoperation among the solo function group that the robot 58 has. Thecontroller 28 may control the notification of the execution solofunction, or may cause the robot 58 to execute the execution solofunction.

Further, as illustrated in FIG. 17, when the device images 60 and 70 aredisplayed on the extension line 80, the controller 28 specifies therobot 58 associated with the device image 60 and the multifunctiondevice 68 associated with the device image 70 as device imagesassociated with the control targets (objects to be linked). Thespecifying unit 30 identifies the robot 58 associated with the deviceimage 60 and the multifunction device 68 associated with the deviceimage 70 as the control targets (objects to be linked). Next, byreferring to the linkage function management table illustrated in FIG.7, the specifying unit 30 specifies the execution linkage functionassociated with the above-described bending operation among the linkagefunction group executable using the robot 58 and the multifunctiondevice 68. The controller 28 may control the notification of theexecution linkage function, or may cause the robot 58 and themultifunction device 68 to execute the execution linkage function.

The user may operate the UI unit 20 to change the display position ofthe device images 60 and 70 such that the device images 60 and 70 aredisplayed on the extension line 80. For example, as illustrated in FIG.17, the user may change the display positions of the device images 60and 70 such that the device images 60 and 70 are aligned in a straightline, and then bend the end portion 76 in the direction of the arrow 78.Thereby, even if the device images 60 and 70 are not displayed on theextension line 80 before changing the display positions of the deviceimages 60 and 70, the device images 60 and 70 are displayed on theextension line 80. Thus, the robot 58 and the multifunction device 68are identified as objects to be linked.

According to Example 3, even in a case where plural device images aredisplayed, when one device image is displayed on the extension line inthe operation direction, the device associated with the one device image(a single device) is identified as a control target, and notification orexecution of the execution solo function that the device has iscontrolled. Further, in a case where plural device images are displayedon the extension line in the operation direction, the devices associatedwith the respective device images are identified as control targets(objects to be linked), and notification or execution of the executionlinkage function executable using the plural devices is controlled.Thereby, it is possible to switch between the solo function and thelinkage function according to the deformation operation on the flexibledisplay.

Example 4

Example 4 will be described. In Example 4, the controller 28 changes anexecution linkage function according to the arrangement order of pluraldevice images. For example, the controller 28 changes the executionlinkage function according to the arrangement order of the plural deviceimages displayed in the direction of the operation, starting from aplace where an operation is performed on the flexible display.Hereinafter, Example 4 will be described in detail.

FIG. 18 illustrates an exemplary linkage function management tableaccording to Example 4. In the linkage function management table, as anexample, information indicating a combination of device IDs, informationindicating the names of devices to be linked (e.g., the types ofdevices), information indicating execution linkage functions (executionlinkage function information), information indicating an arrangementorder, and information indicating a priority are associated with eachother. The device ID and the device name are examples of the deviceidentification information. A linkage function executable using three ormore devices may be specified. In this case, a combination of three ormore devices and a linkage function are associated with each other, andthe association is registered in the linkage function management table.

The arrangement order corresponds to an arrangement order of the deviceimages displayed in the direction of the operation performed on theflexible display.

The priorities are priorities of notification of information on theexecution linkage functions (e.g., display or voice output). Forexample, a combination of PC (A) and multifunction device (B) isassociated with a “scan transfer function” and a “print function” aslinkage functions. For example, when the respective device images aredisplayed in the arrangement order of the device image associated withthe multifunction device (B) and the device image associated with the PC(A) with respect to the operation direction, the priority of the “scantransfer function” is “1st,” and the priority of the “print function” is“2nd.” In this case, the information on the “scan transfer function” isnotified in preference to (for example, at a higher rank than) theinformation on the “print function.” Conversely, when the respectivedevice images are displayed in the arrangement order of the device imageassociated with the PC (A) and the device image associated with themultifunction device (B) with respect to the operation direction, thepriority of the “print function” is “1st,” and the priority of the “scantransfer function” is “2nd.” In this case, the information on the “printfunction” is displayed in preference to (for example, at a higher rankthan) the information on the “scan transfer function.” Of course, thecontroller 28 may cause the plural devices to be linked to execute theexecution linkage function having “1st” priority, along with notifyingthe information on the execution linkage function or without notifyingthe information.

For example, in the example illustrated in FIG. 17, the end portion 76is a starting point of the operation on the flexible display. On thebasis of the position of the starting point, the respective deviceimages are displayed on the extension line 80 in the direction of theoperation in the arrangement order of the device images 60 and 70. Byreferring to the linkage function management table illustrated in FIG.18, the specifying unit 30 specifies an execution linkage functionassociated with the device images 60 and 70 arranged in this order(e.g., an execution linkage function A), which is a linkage functionexecutable using the robot 58 and the multifunction device 68, as anexecution linkage function having “1st” priority. Further, thespecifying unit 30 specifies another execution linkage function (e.g.,an execution linkage function B) as an execution linkage function having“2nd” priority. The controller 28 notifies the execution linkagefunction A in preference to the execution linkage function B (forexample, displays the execution linkage function A at a higher rank).The controller 28 may cause the robot 58 and the multifunction device 68to execute the execution linkage function A.

FIG. 19 illustrates another example. In the example illustrated in FIG.19, the respective device images are displayed on the extension line 80in the direction of the operation in the arrangement order of the deviceimages 70 and 60, on the basis of the position of the starting point. Byreferring to the linkage function management table illustrated in FIG.18, the specifying unit 30 specifies an execution linkage functionassociated with the device images 70 and 60 arranged in this order(e.g., the execution linkage function B), which is a linkage functionexecutable using the robot 58 and the multifunction device 68, as anexecution linkage function having “1st” priority. Further, thespecifying unit 30 specifies another execution linkage function (e.g.,the execution linkage function A) as an execution linkage functionhaving “2nd” priority. The controller 28 notifies the execution linkagefunction B in preference to the execution linkage function A (forexample, displays the execution linkage function A at a higher rank).The controller 28 may cause the robot 58 and the multifunction device 68to execute the execution linkage function B.

The user may operate the UI unit 20 to change the arrangement order ofthe device images 60 and 70. As a result, even when the same deformationoperation is performed, the priorities of the execution linkagefunctions may be changed.

According to Example 4, the notification order (e.g., display order) ofinformation on the execution linkage functions and execution thereof arechanged according to the arrangement order of the device images withrespect to the direction of the deformation operation. Thus,notification and execution of various linkage functions may be performedby changing the arrangement order and the direction of the deformationoperation.

For example, the arrangement order of the device images may also serveas an order of functions used in each device or an order of movement ofdata moving between the plural linked devices. Therefore, by changingthe notification order and execution of the execution linkage functionsaccording to the arrangement order with respect to the operationdirection, the execution linkage function predicted to be used by theuser is preferentially notified or executed. That is, the executionlinkage function which is more likely to be used by the user is morepreferentially notified or displayed. For example, in the case whererespective devices are displayed in the arrangement order of the deviceimage associated with multifunction device (B) and the device imageassociated with the PC (A) on the extension line of the operationdirection, it is predicted that the user would use the linkage functionof “using the function of the multifunction device (B) prior to the PC(A) and transferring the data from the multifunction device (B) to thePC (A).” Conversely, in the case where respective devices are displayedin the arrangement order of the device image associated with the PC (A)and the device image associated with the multifunction device (B), it ispredicted that the user would use the linkage function of “using thefunction of the PC (A) prior to the multifunction device (B) andtransferring the data from the PC (A) to the multifunction device (B).”Therefore, by changing the notification order and execution of theexecution linkage functions according to the arrangement order of thedevice images, the execution linkage function which is more likely to beused by the user is preferentially notified or executed.

Example 5

Example 5 will be described. In Example 5, different solo functions areassigned to respective portions in the device image associated with thedevice. The specifying unit 30 specifies a solo function executable byusing a portion of the device associated with the device image displayedat the changed position on the flexible display. Hereinafter, Example 5will be described in detail.

FIG. 20 illustrates an exemplary device function management tableaccording to Example 5. In the device function management table, as anexample, device IDs, information indicating the names of devices (e.g.,the types of devices), information indicating a position (position of aportion) in a device image, information indicating a solo functionassigned to the position (position of the portion) in the device image(solo function information), and image IDs are associated with eachother. The position in the device image is a specific position (specificportion) in the device image associated with the device. For example,the position in the device image is a specific position in the deviceimage schematically representing the device or a specific position in adevice image captured by the camera. Different functions are assigned torespective positions (portions) in the device image.

Hereinafter, portions of a device will be described with reference toFIG. 21. As illustrated in FIG. 21, the device image 70 associated withthe multifunction device 68 is displayed on the flexible display. Themultifunction device 68 is identified by using the identificationtechnology (e.g., AR technology) as described above. Further, eachportion of the multifunction device 68 is also identified by using theabove identification technology. For example, each portion of the deviceis captured, and each portion is identified using the markerless ARtechnology.

For example, a “print function” is assigned to a position (portion image70 a) associated with a main body portion of the multifunction device 68in the device image 70. A “scan function” is assigned to a position(portion image 70 b) associated with a document cover, a document glass,or an automatic document feeder of the multifunction device 68 in thedevice image 70. A “stapling function” is assigned to a position(portion image 70 c) associated with a post-processing device in thedevice image 70. The “stapling function” is a function of stapling theoutput paper.

The user may separate each portion image from the device image foroperation. For example, when the user selects the portion image 70 b andseparates the portion image 70 b from the device image 70 while thedevice image 70 is displayed, the controller 28 displays the portionimage 70 b away from the device image 70 on the flexible display. Forexample, the user may operate a portion image by a drag-and-dropoperation. Specifically, the user may select a portion image, separatethe portion image from the device image by a drag operation, and performa drop operation at a target display position, so that the portion imagemay be displayed at the display position. Of course, even when theoperation for separating the portion image from the device image isperformed, the portion image may be displayed at the original position.

For example, as illustrated in FIG. 21, the end portion 76 is bent inthe direction of the arrow 78, and the portion image 70 a included inthe device image 70 is displayed on the extension line 80 in thedirection of the bending operation. By referring to the device functionmanagement table illustrated in FIG. 20, the specifying unit 30specifies a solo function (e.g., a print function) associated with theposition of the portion image 70 a in the device image 70. Thecontroller 28 may notify the print function as the execution solofunction or may cause the multifunction device 68 to execute the printfunction.

In a case where plural portion images (e.g., the portion images 70 a and70 b) are displayed on the extension line 80, the specifying unit 30refers to the device function management table illustrated in FIG. 20and specifies solo functions (e.g., the print function and the scanfunction) associated with the positions of the respective portion imagesin the device image. The controller 28 may notify the print function andthe scan function as the execution solo functions, respectively, or maycause the multifunction device 68 to execute the print function or thescan function. As another example, the specifying unit 30 may specify alinkage function executable using the solo functions associated with therespective portion images, and the controller 28 may notify or executethe linkage function.

According to Example 5, when a device as a control target has pluralsolo functions, solo functions are individually designated, and thedesignated solo functions are notified or executed.

The user may operate the UI unit 20 to change the entire displayposition of the device image or separate the portion image from thedevice image, thereby displaying the portion image desired by the useron the extension line 80.

Example 6

Example 6 will be described. In Example 6, as in Example 5, differentsolo functions may be assigned to respective portions in the deviceimage associated with the device. The specifying unit 30 specifies alinkage function executable by using the portion images associated withthe respective portion images displayed at the changed positions on theflexible display. Hereinafter, Example 6 will be described in detail.

FIG. 22 illustrates an exemplary device function management tableaccording to Example 6. In the device function management table, as anexample, a device ID, information indicating the device name (e.g., thetype of the device), a portion ID as portion identification informationfor identifying the portion, information indicating a solo functionassigned to the portion (a solo function that the portion has), and aportion image ID for identifying the portion image associated with theportion are associated with each other. The portion image is an imagerepresenting the external appearance of the portion of the deviceobtained by capturing by the camera. Of course, the portion image thatschematically representing a portion of the device may be associatedwith the portion. For example, different functions are assigned to therespective portions of the device.

To describe with a specific example, a screen display function isassigned to the display unit of the PC (A), and information indicatingthe screen display function is associated with the portion image ID ofthe portion image associated with the display unit. The screen displayfunction is a function of displaying information on the PC (A). A datasave function is assigned to the main body portion of the PC (A), andinformation indicating the data save function is associated with theportion image ID of the portion image associated with the main bodyportion. The data save function is a function of saving (storing) datain the PC (A).

Further, a print function is assigned to the main body portion of themultifunction device (B), and information indicating the print functionis associated with the portion image ID of the portion image associatedwith the main body portion. A scan function is assigned to the readingunit of the multifunction device (B) (e.g., a portion associated withthe document cover, the document glass, or the automatic document feederof the multifunction device (B)), and information indicating the scanfunction is associated with the portion image ID of the portion imageassociated with the reading unit. A stapling function is assigned to thepost-processing device of the multifunction device (B), and informationindicating the stapling function is associated with the portion image IDof the portion image associated with the post-processing device.

A solo function assigned to a portion of a device is specified(identified), for example, by using the markerless AR technology. Forexample, when a portion of a device is captured by a camera (e.g., thecamera 22 of the terminal device 10), the specifying unit 30 specifies(identifies) a solo function associated with the external appearanceimage data in the device function management table illustrated in FIG.22. Thereby, the solo function assigned to the captured portion isspecified (identified). For example, when the main body portion of themultifunction device (B) is captured by the camera 22, the specifyingunit 30 specifies the print function associated with the externalappearance image data representing the main body portion of themultifunction device (B) in the device function management table.Thereby, it is specified that the function assigned to the main bodyportion of the multifunction device (B) is the print function.

Of course, a solo function assigned to a portion of a device may bespecified (identified) by using the marker AR technology. For example, amarker such as a two-dimensional barcode in which portion identificationinformation (e.g., a portion ID) for identifying a portion is coded isprovided in each portion of the device. When the marker provided in theportion is captured by the camera and the marker AR technology isapplied, the portion identification information (e.g., the portion ID)of the portion is acquired. When the portion identification informationis thus acquired, the specifying unit 30 specifies (identifies) the solofunction associated with the portion identification information (e.g.,the portion ID) in the device function management table illustrated inFIG. 22.

FIG. 23 illustrates an exemplary linkage function management tableaccording to Example 6. The linkage function management table isinformation indicating a linkage function that uses solo functions thatplural portions have. In the linkage function management table, as anexample, information indicating a combination of portions of devices,information indicating a combination of portion IDs, and informationindicating an execution linkage function executable using the solofunctions that the plural portions included in the combination have areassociated with each other. Of course, in the linkage functionmanagement table, information indicating a combination of a portion of adevice and an entire device, and information indicating a linkagefunction using a solo function that the portion of the device has and asolo function that the entire device has may be associated with eachother.

To describe with a specific example, a “print function” as a linkagefunction is assigned to a combination of the display unit of the PC (A)and the main body portion of the multifunction device (B). The “printfunction” as a linkage function is a function of transmitting datastored in the PC (A) to the multifunction device (B) and printing thedata by the multifunction device (B).

Further, a “print function” as a linkage function is assigned to acombination of the main body portion of the multifunction device (B) andthe main body portion of a projector (C). The “print function” as alinkage function is a function of transmitting data projected by theprojector (C) to the multifunction device (B) and printing the data bythe multifunction device (B).

Further, a “scan projection function” as a linkage function is assignedto a combination of the reading unit of the multifunction device (B) andthe main body portion of a projector (C). The scan projection functionas a linkage function is, for example, a function of transmitting datagenerated by scanning by the multifunction device (B) to the projector(C) and projecting the data by the projector (C).

The linkage function may be a function executable using solo functionsthat plural portions included in the same device have or a functionexecutable using solo functions that portions of plural differentdevices have. Further, the linkage function may be a function executableusing solo functions that three or more portions have.

Example 6 will be described with a specific example with reference toFIG. 24. As illustrated in FIG. 24, the device image 70 associated withthe multifunction device 68 and a device image 82 associated with the PC(A) are displayed on the flexible display. The multifunction device 68and the PC (A) are identified by the above identification technology.The device image 82 includes a portion image 82 a associated with themain body portion of the PC (A) (including a processor such as a CPU, astorage device such as an HDD and an SSD, other boards, etc.) and aportion image 82 b associated with a display of the PC (A).

For example, the end portion 76 is bent in the direction of the arrow78, and the portion image 70 a included in the device image 70 and theportion image 82 b included in the device image 82 are displayed on theextension line 80 in the direction of the bending operation. Byreferring to the linkage function management table illustrated in FIG.23, the specifying unit 30 specifies an execution linkage functionassociated with the combination of the portion associated with theportion image 70 a and the portion associated with the portion image 82b. The controller 28 may notify the execution linkage function, or maycause the multifunction device 68 and the PC (A) to execute theexecution linkage function.

According to Example 6, when a device as a control target has pluralsolo functions, the plural solo functions are individually designated,and a linkage function using the designated plural solo functions arenotified or executed.

The linkage function may be a function that uses a combination ofportions of the same device or a combination of portions of differentdevices, or may be a function that uses a combination of the entiredevice and a portion of the device.

Further, by combining Examples 4 and 6, the linkage function may bechanged according to the arrangement order of the device images or theportion images with respect to the operation direction.

The user may operate the UI unit 20 to change the entire displayposition of the device image or separate the portion image from thedevice image, thereby displaying the portion image desired by the useron the extension line 80. As a result, even when the same deformationoperation is performed, various linkage functions are notified andexecuted.

Example 7

Example 7 will be described. In Example 7, the detection unit 24 detectschanges in the respective screens of the flexible display. For example,when the flexible display has the front side display 40 as a firstscreen and the rear side display 42 as a second screen, the detectionunit 24 detects respective changes in the front side display 40 and therear side display 42. When a device as a control target is displayed onthe flexible display and when the detection unit 24 detects a validchange with respect to the front side display 40, the specifying unit 30specifies a solo function associated with the front side of the device.Meanwhile, when the detection unit 24 detects a valid change withrespect to the rear side display 42, the specifying unit 30 specifies asolo function associated with the rear side of the device. Thecontroller 28 notifies or executes the solo function specified by thespecifying unit 30. For example, when a bending operation of theflexible display is performed toward the front side display 40 side, thedetection unit 24 detects a change due to the bending operation as avalid change with respect to the front side display 40. Meanwhile, whena bending operation of the flexible display is performed toward the rearside display 42 side, the detection unit 24 detects a change due to thebending operation as a valid change with respect to the rear sidedisplay 42. Hereinafter, Example 7 will be described in detail.

FIG. 25 illustrates the rear side display 42 of the flexible display(terminal device 10). A rear side image 84 representing the rear side ofthe robot 58 as a control target is displayed on the rear side display42. As illustrated in FIG. 8, a front side image (an image displayed asthe device image 60) representing the front side of the robot 58 isdisplayed on the front side display 40.

For example, as illustrated in FIG. 8, when the user bends the entireleft side portion 10 a of the flexible display (the terminal device 10itself) to the front side display 40 side toward the right side portion10 b of the flexible display, the detection unit 24 detects a change inthe flexible display due to the bending operation as a valid change withrespect to the front side display 40. In this case, the specifying unit30 specifies the solo function associated with the front side of therobot 58 displayed on the front side display 40. That is, the same ordifferent solo functions are assigned to the front side and rear side ofthe robot 58, respectively, and the information indicating theassignment is stored in the terminal device 10, the server 16, or thelike as the device function management information. By referring to thedevice function management information, the specifying unit 30 specifiesthe solo function associated with the front side. For example, the solofunction is a function that the front side portion of the robot 58 has.

Meanwhile, as illustrated in FIG. 25, when the user bends the entireright side portion 10 b of the flexible display to the rear side display42 side toward the left side portion 10 a of the flexible display (in acase of bending in the direction of the arrow 86), the detection unit 24detects a change in the flexible display due to the bending operation asa valid change with respect to the rear side display 42. In this case,by referring to the above-described device management information, thespecifying unit 30 specifies the solo function associated with the rearside of the robot 58 displayed on the rear side display 42. For example,the solo function is a function that the rear side portion of the robot58 has.

The controller 28 notifies or executes the solo function specified bythe specifying unit 30.

According to Example 7, various functions according to the deformationoperation on the flexible display having the plural display units arenotified or executed.

Further, Example 7 may be applied to specifying the linkage function.For example, when the flexible display is bent toward the front sidedisplay 40, the specifying unit 30 specifies a linkage functionexecutable using solo functions that portions of devices displayed onthe front side display 40 have (e.g., solo functions that the front sideportions have). Further, when the flexible display is bent toward therear side display 42, the specifying unit 30 specifies a linkagefunction executable using solo functions that portions of devicesdisplayed on the rear side display 42 have (e.g., solo functions thatthe rear side portions have).

Further, by combining Examples 4 and 7, the linkage function may bechanged according to the arrangement order of the device images.

(Connection Process During Linkage)

Here, an operation when executing a linkage function will be described.For example, when a linkage function is executed, a connection requestis transmitted from the terminal device 10 to each device to be linked,and communication is established between the terminal device 10 and eachdevice. Hereinafter, the connection process will be described withreference to FIG. 26. FIG. 26 is a sequence diagram illustrating theprocess.

When the robot 58 and the multifunction device 68 are identified asdevices to be linked, the terminal device 10 transmits informationindicating the connection request to the devices to be linked (the robot58 and the multifunction device 68) (S01). For example, when addressinformation indicating the address of each of the devices to be linkedis stored in the server 16, the terminal device 10 acquires the addressinformation of each of the devices to be linked from the server 16. In acase where the address information is included in the deviceidentification information, the controller 28 of the terminal device 10acquires the address information from the device identificationinformation of each device. Of course, the address information of eachdevice may be stored in the storage unit 26 of the terminal device 10.The terminal device 10 transmits the information indicating theconnection request to each of the devices to be linked (e.g., the robot58 and the multifunction device 68) by using the address information ofeach of the devices to be linked.

The robot 58 and the multifunction device 68 that receive theinformation indicating the connection request determine whether topermit the connection with the terminal device 10 (S02). For example,when the robot 58 and the multifunction device 68 are devices that arenot permitted to connect, or when the number of devices requestingconnection exceeds the upper limit, connection is not permitted. Whenthe connection from the terminal device 10 is permitted, in order toprevent that setting information unique to each of the robot 58 and themultifunction device 68 is not changed from the terminal device 10, thechanging operation may be prohibited. For example, it may be prohibitedto change the color parameters of the multifunction device 68, thesetting time when shifting to power saving, and the like. Thereby, thesecurity for the device to be linked is improved. As another example,when linking devices, change of setting information may be restricted ascompared with a case where the devices are used alone without beinglinked. For example, it may be permitted to change the smaller number ofsetting items as compared with the case where the device is used alone.In addition, browsing of other users' personal information such as theoperation history may be prohibited. As a result, the security for theuser's personal information is improved.

Result information indicating permission or non-permission of theconnection is transmitted from the robot 58 and the multifunction device68 to the terminal device 10 (S03). When the connection to the robot 58and the multifunction device 68 is permitted, communication isestablished between the terminal device 10 and the robot 58, and furthercommunication is established between the terminal device 10 and themultifunction device 68.

Next, in the terminal device 10, the user instructs execution of alinkage function (S04). The terminal device 10 transmits informationindicating the execution instruction of the linkage function (executioninstruction information) to each of the devices to be linked (the robot58 and the multifunction device 68) according to the instruction (S05).The execution instruction information transmitted to the robot 58includes information indicating a process to be executed by the robot58, and the execution instruction information transmitted to themultifunction device 68 includes information indicating a process to beexecuted by the multifunction device 68.

When receiving the execution instruction information, the robot 58 andthe multifunction device 68 execute the linkage function according tothe execution instruction information (S06). When the linkage functionto be executed includes a process of transmitting and receiving databetween the robot 58 and the multifunction device 68, communication isestablished between the robot 58 and the multifunction device 68. Forexample, the execution instruction information transmitted to the robot58 includes the address information of the multifunction device 68, andthe execution instruction information transmitted to the multifunctiondevice 68 includes the address information of the robot 58.Communication is established between the robot 58 and the multifunctiondevice 68 using the address information. After the communication isestablished, the robot 58 and the multifunction device 68 execute thelinkage function while transmitting and receiving data to and from eachother.

When the execution of the linkage function is completed, informationindicating completion of execution of the linkage function istransmitted from the robot 58 and the multifunction device 68 to theterminal device 10 (S07). Information indicating that execution of thelinkage function is completed is displayed on the UI unit 20 of theterminal device 10 (S08). When information indicating completion ofexecution is not displayed even though a preset time elapses since theexecution instruction is given, the controller 28 of the terminal device10 may cause the UI unit 20 to display information indicating an error,and again transmit the execution instruction information or theinformation indicating the connection request to the robot 58 and themultifunction device 68.

Next, the user confirms whether or not to release the linked statebetween the robot 58 and the multifunction device 68 (S09), and aprocess is executed according to the presence or absence of the release(S10). When releasing the linked state, the user gives an instruction ofrelease by using the terminal device 10. Therefore, the communicationbetween the terminal device 10 and the robot 58 is released, and furtherthe communication between the terminal device 10 and the multifunctiondevice 68 is released. Similarly, the communication between the robot 58and the multifunction device 68 is released as well. When the linkedstate is not to be released, the execution instruction may be givencontinuously.

Further, the number of devices to be linked may be increased. Forexample, a third device may be designated as an object to be linked by adeformation operation, and the three devices including the robot 58 andthe multifunction device 68 may be linked. Information indicating thatthe robot 58 and the multifunction device 68 are already identified asobjects to be linked is stored in the terminal device 10 or the server16.

The device identification information indicating the device to be linkedand the linkage function information indicating the executed linkagefunction may be stored in the terminal device 10 or the server 16. Forexample, user account information (an example of user identificationinformation) of the user who uses the terminal device 10 may beacquired, and history information indicating an association among theuser account information, the device identification informationindicating the devices to be linked, and the linkage functioninformation indicating the executed linkage function may be created andstored in the terminal device 10 or the server 16. The historyinformation may be created by the terminal device 10 or may be createdby the server 16. By referring to the history information, it isspecified what linkage function is executed using what device group.

The devices to be linked (e.g., the robot 58 and the multifunctiondevice 68) may store the user account information of the user whorequests the connection and the terminal identification informationindicating the terminal device 10 of the connection request source asthe history information. The user using the device is specified byreferring to this history information. For example, the user may bespecified by utilizing the history information in the case ofidentifying the user who uses the device when the device is broken or inthe case of performing the charging process on the expendable item. Thehistory information may be stored in the terminal device 10 or theserver 16 or may be stored in another device.

The user account information is stored, for example, in the storage unit26 of the terminal device 10 in advance, and the controller 28 of theterminal device 10 functions as an example of a user identificationunit, reads the user account information of the user from the storageunit 26, and identifies the user who uses the terminal device 10. Whenthe user account information of plural users is stored in the storageunit 26, the user designates his/her own user account information usingthe terminal device 10. As a result, the user account information of theuser is read and the user is identified. As another example, thecontroller 28 of the terminal device 10 may identify the user by readingthe user account information of the user logged in to the terminaldevice 10. As still another example, when only one user accountinformation is stored in the same terminal device 10, the controller 28of the terminal device 10 may identify the user by reading the useraccount information. When the user account is not set and the useraccount information is not created, the initial setting is performed, sothat the user account information is created.

The use history of the cooperative function may be managed for eachuser, and information indicating the linkage function used in the pastby the user indicated by the read user account information may bedisplayed on the UI unit 20 of the terminal device 10. The informationindicating the use history may be stored in the terminal device 10 ormay be stored in the server 16. In addition, information indicating alinkage function used at a frequency equal to or higher than a presetuse frequency may be displayed. The time and effort of the user'soperation related to the linkage function is reduced by providing such ashortcut function.

Further, when the linkage function associated with a combination ofplural devices designated by the deformation operation is not registeredin the linkage function management table, the specifying unit 30determines that no usable linkage function is registered. In this case,the controller 28 causes the display unit of the UI unit 20 to displayinformation indicating that no usable linkage function is registered.Even in such a case, the linkage function may be usable depending on theupdate status of the device or the like. In this case, the specifyingunit 30 specifies the linkage function that becomes usable.

Second Exemplary Embodiment

Hereinafter, an information processing system according to a secondexemplary embodiment will be described. In the first exemplaryembodiment, the control target is a device, but in the second exemplaryembodiment, the control target is a function. For example, when afunction image associated with a device is displayed on the flexibledisplay and when a deformation operation is performed on the flexibledisplay, a function associated with the function image displayed at thechanged position is identified as a control target. When one functionimage is displayed at the changed position, a solo function associatedwith the function image is identified as a control target. When pluralfunction images are displayed, solo functions associated with therespective function image are identified as control targets (objects tobe linked).

The function image may be, for example, an image associated withsoftware (an application, etc.) or an image associated with a devicewhich performs a process (a device present in the real space). Thefunction image is, for example, an image such as an icon, which may beregarded as a virtual image in the sense of distinguishing from an imagerepresenting an actual external appearance of a device as hardware(e.g., a photograph) and an image representing the device itself.

Hereinafter, specific examples of the second exemplary embodiment willbe described in detail.

Example 8

Example 8 will be described. FIG. 27 illustrates the terminal device 10.A function image 88 is displayed on the front side display 40 of theterminal device 10. The function image 88 is an image associated with apassword setting function (e.g., an image associated with passwordsetting software). When the user selects the function image 88 and givesan instruction to activate the password setting function, for example,the password setting software is activated, and the password settingprocess is performed on data or the like. Further, informationindicating the name of the function may be displayed. For example, thecharacter string “password setting function” may be displayed togetherwith the function image 88. Of course, the function image 88 may not bedisplayed, but the character string may be displayed. The softwareassociated with the function image may be installed in the terminaldevice 10 or may be stored in an external device such as the server 16,and the function by the software may be provided from the externaldevice to the terminal device 10. The user may operate the UI unit 20 tochange the display position and display size of the function image 88.

For example, it is assumed that the user bends the whole of the leftside portion 10 a of the flexible display (terminal device 10 itself)toward the right side portion 10 b of the flexible display. That is, itis assumed that the user bends the flexible display in the direction ofan arrow 89 illustrated in FIG. 27.

The detection unit 24 detects the change (deformation) of the flexibledisplay due to the above-described bending operation, specifies theposition where the change occurs, and the controller 28 specifies thefunction image displayed at that position. When a change occurs at theposition where the function image 88 is displayed due to theabove-described bending operation, the specifying unit 30 identifies thepassword setting function associated with the function image 88 as acontrol target. When a character string indicating a function isdisplayed in place of the function image, the controller 28 may specifythe character string displayed at the changed position, and thespecifying unit 30 may identify a function associated with the characterstring (e.g., the password setting function) as a control target. Thecontroller 28 may notify the user of the status that the passwordsetting function is designated (display information indicating thestatus or output the status as voice), or may execute the passwordsetting function. Further, the controller 28 may change the functionimage 88. For example, the controller 28 may change the function image88 to an image representing the designated state, or may change thefunction image 88 to an image representing a state where the passwordsetting function is being executed.

Similarly to Example 1 described above, the execution solo function maybe assigned to each deformation operation. In this case, the specifyingunit 30 specifies an execution solo function associated with the bendingoperation, and the controller 28 notifies or executes the execution solofunction. Further, the controller 28 may change the execution solofunction according to the mode of the deformation operation.

According to Example 8, when the user deforms the flexible display, thefunction associated with the function image displayed at the changedposition is notified or executed. Thereby, the function is notified orexecuted by a simple operation.

Further, similarly to Example 3, the specifying unit 30 may identify thefunction associated with the function image displayed on the extensionline in the direction of the deformation operation as a control target,and the controller 28 may notify or execute the function.

Further, the same or different functions may be assigned to each portionin the function image. In this case, the portion displayed at thechanged position of the flexible display is identified, and the functionassigned to that portion is notified or executed.

Example 9

Example 9 will be described. FIG. 28 illustrates an exemplary linkagefunction management table according to Example 9. The linkage functionmanagement table is information indicating linkage functions executableusing plural solo functions. In the linkage function management table,as an example, function IDs (e.g., IDs for identifying software),information indicating the function names (e.g., the name of software),information indicating linkage functions executable using plural solofunctions, deformation operation information indicating the deformationoperation performed on the flexible display, and execution linkagefunction information indicating a linkage function to be executed whenthe deformation operation is performed, among a linkage function groupexecutable using plural solo functions to be linked are associated witheach other. A linkage function executable using plural solo functions isspecified by referring to the linkage function management table.

For example, software a as a function is software that transmits data,and software β as a function is software that sets a password for datawith a password. For example, a linkage function of setting a passwordto data and then transmitting the data is implemented by combining thesoftware a and the software β. For other software, an executable linkagefunction is similarly specified. A linkage function executable usingthree or more solo functions (e.g., software) may be specified. In thiscase, a combination of three or more solo functions and a linkagefunction are associated with each other, and the association isregistered in the linkage function management table.

A combination of the plural solo functions is associated with one ormore deformation operations, and the respective deformation operationsare associated with the same or different execution linkage functions.

Hereinafter, Example 9 will be described in detail with reference toFIG. 29. FIG. 29 illustrates the terminal device 10. Function images 88and 90 are displayed on the front side display 40 of the terminal device10. The function image 90 is an image associated with a datatransmission function (e.g., an image associated with data transmissionsoftware). When the user selects the function image 90 and gives aninstruction to activate the data transmission function, for example, thedata transmission software is activated, and data to be transmitted andthe like are transmitted to a transmission destination. The user mayoperate the UI unit 20 to change the display positions and display sizesof the function images 88 and 90. Information indicating the names ofthe password setting function and the data transmission function (e.g.,character strings) may be displayed along with the function images 88and 90 or in place of the function images 88 and 90.

For example, it is assumed that the user bends the whole of the leftside portion 10 a of the flexible display (terminal device 10 itself)toward the right side portion 10 b of the flexible display. That is, itis assumed that the user bends the flexible display in the direction ofan arrow 92.

The detection unit 24 detects the change (deformation) of the flexibledisplay due to the above-described bending operation, specifies theposition where the change occurs, and the controller 28 specifies thefunction image displayed at that position. When changes occur at thepositions where the function images 88 and 90 are displayed due to theabove-described bending operation, the specifying unit 30 identifies thepassword setting function associated with the function image 88 and thedata transmission function associated with the function image 90 ascontrol targets (objects to be linked) and refers to the linkagefunction management table illustrated in FIG. 28 to specify theexecution linkage function associated with the deformation operationamong the linkage function group executable using the password settingfunction (password setting software) and the data transmission function(data transmission software). For example, it is assumed that theexecution linkage function is a function of setting a password to datato be processed and transmitting the data to a destination.

For example, the controller 28 causes the flexible display to displayinformation indicating the execution linkage function specified asdescribed above. The controller 28 may output information indicating theexecution linkage function as voice information. When the user gives aninstruction to execute the execution linkage function, the controller 28activates the password setting software to set the password for the datato be processed, and further activates the data transmission software totransmit the data to the destination. In the case where theabove-described bending operation is performed, the controller 28 mayexecute the execution linkage function even when the executioninstruction is not given by the user. Further, the controller 28 maychange the function images 88 and 90.

According to Example 9, when the user deforms the flexible display, thelinkage function according to the deformation is notified or executed.Thereby, the linkage function is notified or executed by a simpleoperation.

Further, similarly to Example 3, the specifying unit 30 may identifyfunctions associated with the respective function images displayed onthe extension line in the direction of the deformation operation asobjects to be linked (control targets), and specify an execution linkagefunction associated with the deformation operation among the linkagefunction group executable using the plural functions. The controller 28notifies or executes the execution linkage function. The controller 28may change the linkage function according to the mode of the deformationoperation.

Further, Example 4 may be applied to Example 9. That is, the linkagefunction may be changed according to the arrangement order of the pluralfunction images with respect to the operation direction. Further, thesame or different functions may be assigned to each portion in thefunction image.

Third Exemplary Embodiment

Hereinafter, an information processing system according to a thirdexemplary embodiment will be described. In the third exemplaryembodiment, the control targets are a device and a function. Forexample, when a device image associated with a device and a functionimage associated with a function are displayed on the flexible displayand when a deformation operation is performed on the flexible display, adevice associated with the device image and a function associated withthe function image displayed at the changed position are identified asobjects to be linked (control targets). Then, the execution linkagefunction associated with the deformation operation is specified amongthe linkage function group executable using the device and the function,and the execution linkage function is notified or executed.

Hereinafter, specific examples of the third exemplary embodiment will bedescribed in detail.

Example 10

Example 10 will be described. FIG. 30 illustrates an exemplary linkagefunction management table according to Example 10. The linkage functionmanagement table is information indicating linkage functions executableusing devices and functions (e.g., software). In the linkage functionmanagement table, as an example, a combination of IDs (a combination ofa device ID and a function ID), information indicating a device name anda function name, linkage function information indicating linkagefunctions executable using a device and a function, deformationoperation information indicating a deformation operation performed onthe flexible display, and execution linkage function informationindicating a linkage function to be executed when the deformationoperation is performed among a linkage function group executable usingthe device and the function which are to be linked are associated witheach other. The linkage function executable using the device and thefunction is specified by referring to the linkage function managementtable. For example, a linkage function of transmitting data using the PC(A) is implemented by combining, for example, the PC (A) as a device Awith the data transmission function as a function a. For other devicesand functions, executable linkage functions are similarly specified. Alinkage function executable using three or more devices and functionsmay be specified. In this case, a combination of three or more devicesand functions and a linkage function are associated with each other, andthe association is registered in the linkage function management table.

A combination of the device and the function is associated with one orplural deformation operations, and the respective deformation operationsare associated with the same or different execution linkage functions.

Hereinafter, Example 10 will be described in detail with reference toFIG. 31. FIG. 31 illustrates the terminal device 10. The device image 70and the function image 88 are displayed on the front side display 40 ofthe terminal device 10.

For example, it is assumed that the user bends the whole of the leftside portion 10 a of the flexible display (terminal device 10 itself)toward the right side portion 10 b of the flexible display. That is, itis assumed that the user bends the flexible display in the direction ofan arrow 94.

The detection unit 24 detects the change (deformation) of the flexibledisplay due to the above-described bending operation, specifiespositions where the change occurs, and the controller 28 specifies thedevice image and the function image displayed at that positions. Whenchanges occur at the positions where the device image 70 and thefunction image 88 are displayed due to the above-described bendingoperation, the specifying unit 30 identifies the multifunction device 68associated with the device image 70 and the password setting functionassociated with the function image 88 as control targets (objects to belinked) and refers to the linkage function management table illustratedin FIG. 30 to specify the execution linkage function associated with thedeformation operation among the linkage function group executable usingthe multifunction device and the password setting function. For example,it is assumed that the execution linkage function is a function ofsetting a password for data generated by scanning by the multifunctiondevice 68.

For example, the controller 28 causes the flexible display to displayinformation indicating the execution linkage function specified asdescribed above. The controller 28 may output information indicating theexecution linkage function as voice information. When the user gives aninstruction to execute the execution linkage function, the controller 28outputs control information to the multifunction device 68 to therebycontrol scanning by the multifunction device 68 and transfer of datagenerated by the scanning to the terminal device 10. As a result, themultifunction device 68 executes scanning, and data generated by thescanning is transferred from the multifunction device 68 to the terminaldevice 10. Further, the controller 28 activates the password settingsoftware and sets a password for the data. In the case where theabove-described bending operation is performed, the controller 28 mayexecute the execution linkage function even when the executioninstruction is not given by the user. Further, the controller 28 maychange the device image 70 and the function image 88.

According to Example 10, when the user deforms the flexible display, thelinkage function according to the deformation is notified or executed.Thereby, the linkage function is notified or executed by a simpleoperation.

Further, similarly to Example 3, the specifying unit 30 may identify adevice and a function which are respectively associated with the deviceimage and the function image which are displayed on the extension linein the direction of the deformation operation, as objects to be linked(control targets), and specify an execution linkage function associatedwith the deformation operation among the linkage function groupexecutable using the device and the function. The controller 28 notifiesor executes the execution linkage function. The controller 28 may changethe linkage function according to the mode of the deformation operation.

Further, Example 4 may be applied to Example 10. That is, the linkagefunction may be changed according to the arrangement order of the deviceimage and the function image with respect to the operation direction.Further, the same or different functions may be assigned to respectiveportions in the device image, or the same or different functions may beassigned to respective portions in the functional image.

Fourth Exemplary Embodiment

Hereinafter, an information processing system according to a fourthexemplary embodiment will be described. In the fourth exemplaryembodiment, the control target is a file. For example, when a file imageassociated with a file is displayed on the flexible display and when adeformation operation is performed on the flexible display, a fileassociated with the file image displayed at the changed position isidentified as a control target. For example, each file is associatedwith software (e.g., application software) for executing a process onthe file. When one file image is displayed at the changed position, afile associated with the file image is identified as a control target.When plural file images are displayed, respective files associated withthe file images are identified as the control targets (objects to belinked). The file image may be, for example, a schematic image (e.g., anicon or the like), or an image such as a photograph.

Hereinafter, specific examples of the fourth exemplary embodiment willbe described.

Example 11

Example 11 will be described. FIG. 32 illustrates the terminal device10. A file image 96 is displayed on the front side display 40 of theterminal device 10. The file image 96 is, for example, an imageassociated with a document file A. Further, a character stringindicating the name of the document file A may be displayed along withthe file image 96 or in place of the file image 96. The user may operatethe UI unit 20 to change the display position and display size of thefile image 96.

For example, it is assumed that the user bends the whole of the leftside portion 10 a of the flexible display (terminal device 10 itself)toward the right side portion 10 b of the flexible display. That is, itis assumed that the user bends the flexible display in the direction ofan arrow 98 illustrated in FIG. 32.

The detection unit 24 detects the change (deformation) of the flexibledisplay caused by the above-described bending operation, and specifiesthe position where the change occurs, and the controller 28 specifiesthe file image displayed at that position. When a change occurs at theposition where the file image 96 is displayed due to the above-describedbending operation, the specifying unit 30 identifies the document file Aassociated with the file image 96 as a control target. The controller 28may notify the user of a status that software (e.g., document editingsoftware) associated with the document file A is designated, or mayactivate the software to display the content of the document file A (thedocument itself) on the display unit of the UI unit 20. Further, thecontroller 28 may change the file image 96. For example, the controller28 may change the file image 96 to an image representing a designatedstate, or may change the file image 96 to an image representing a statewhere the content of the document file A (the document itself) isdisplayed.

Similarly to Example 1 described above, the execution solo function(e.g., software) may be assigned to each deformation operation. In thiscase, the specifying unit 30 specifies an execution solo functionassociated with the bending operation, and the controller 28 notifies orexecutes the execution solo function. Further, the controller 28 maychange the execution solo function according to the mode of thedeformation operation.

According to Example 11, when the user deforms the flexible display, thefile associated with the file image displayed at the changed position isidentified as a control target, and a function associated with the fileis notified or executed. Thereby, the function is notified or executedby a simple operation.

Further, similarly to Example 3, the specifying unit 30 may identify thefile associated with the file image displayed on the extension line inthe direction of the deformation operation as a control target, and thecontroller 28 may notify or execute a function associated with the file.

Example 12

Example 12 will be described. In Example 12, plural files are identifiedas objects to be linked (control targets), and the linkage functions arenotified or executed. The linkage function management informationaccording to Example 12 is information for managing linkage functionsexecuted by linking plural functions to each other. The linkage functionis, for example, a function implemented by linking plural software toeach other. Of course, the linkage function may be a functionimplemented by linking a function that a device as hardware has and afunction implemented by software, or by linking plural devices.

Hereinafter, the linkage function according to Example 12 will bedescribed in more detail. The linkage function is a function to beexecuted with respect to plural files (data) when the plural files areidentified as objects to be linked (control targets). For example, eachfile is associated with software (application software) for executing aprocess on the file. The linkage function is a function implemented bylinking applications which are associated with the plural files,respectively.

Examples of the file include a document file, an image file, a videofile, a spreadsheet file, a presentation file, a graphic (drawing) file,and an audio file.

The document file is a file having a document format. Also, the documentfile is a file configured with character strings and other pieces ofinformation. The document file may include images, tables, or graphicsas other pieces of information. The document format is, for example, atext format or a document format. For example, data having a text format(e.g., data appended with a character string “.txt” as an extension) anddata having a document format (e.g., data appended with a characterstring “.doc” or “.docx” as an extension) fall into the document files.

The image file is a file having image formation (an image format). Theimage file may be image data having a raster format or image data havinga vector format. The image format is, for example, a JPEG format, a BMP(bitmap) format, a PDF format, a GIF format, a TIFF format, a PNG formator a PICT format. For example, data having a JPEG format (data appendedwith a character string “.jpeg” as an extension), data having a BMPformat (data appended with a character string “.bmp” as an extension),data having a PDF format (data appended with a character string “.pdf”as an extension), data having a GIF format (data appended with acharacter string “.gif” as an extension), data having a TIFF format(data appended with a character string “.tiff” as an extension), datahaving a PNG format (data appended with a character string “.png” as anextension), and data having a PICT format (data appended with acharacter string “.pict” as an extension) fall into the image files. Ofcourse, image data having a format other than these formats may be usedas an image file.

The video file is a file having a video format. The video format is, forexample, an AVI format, an MP4 format, an MOV format, an MPEG2-TSformat, an MPEG2-PS format, a WMV format, or an FLV format. For example,data having an AVI format (data appended with a character string “.avi”as an extension), data having an MP4 format (data appended with acharacter string “.mp4” as an extension), data having an MOV format(data appended with a character string “.mov” as an extension), datahaving an MPEG2-TS format (data appended with a character string “.m2ts”as an extension), data having an MPEG2-PS format (data appended with acharacter string “.mpeg” as an extension), data having a WMV format(data appended with a character string “.wmv” as an extension), and datahaving an FLV format (data appended with a character string “.flv”) fallinto the video files. Of course, video data having a format other thanthese formats may be used as a video file.

The spreadsheet file is a file having a table format. Also, thespreadsheet file is a file configured with tables, graphs, or otherpieces of information. The spreadsheet file may include characterstrings, images, or graphics as other pieces of information. Forexample, the spreadsheet file is data appended with a character string“.csv,” “.xls” or “.xlsx” as an extension.

The presentation file is a file having a presentation sheet format. Forexample, data appended with a character string “.ppt” or “.pptx” as anextension falls into the presentation file. The presentation file isconfigured with, for example, character strings, images (still images ormoving images), graphics, and voice.

The graphic file is a file having a graphic (drawing) format. Thegraphic file is created by, for example, graphic generating software(e.g., two-dimensional CAD or three-dimensional CAD). For example, thegraphic file is data having a DXF format (data appended with a characterstring “.dxf” as an extension) or data having a DWG format (dataappended with a character string “.dwg” as an extension). In addition,data appended with a character string “.vsd” or “.vsdx” as an extensionmay be treated as the graphic file.

The audio file is a file having an audio format. The audio format is,for example, an uncompressed audio format (e.g., a WAV format, an AIFFformat, or a BWF format), a lossy compressed audio format (e.g., an mp3format, an AAC format, or a WMA format), or a lossless compressed audioformat (e.g., a TAK format or an FLAC format). Data having anuncompressed audio format (data appended with, for example, a characterstring “.wav” as an extension), data having a lossy compressed audioformat (data appended with, for example, a character string “.mp3” asextension), and data having an uncompressed audio format (data appendedwith, for example, a character string “.tak” as an extension) fall intothe audio files.

Hereinafter, the linkage function management table according to Example12 will be described with reference to FIG. 33. FIG. 33 illustrates anexemplary linkage function management table according to Example 12. Inthe linkage function management table, as an example, informationindicating a combination of plural file formats, information indicatinglinkage functions (linkage function information), deformation operationinformation indicating the deformation operation performed on theflexible display, and execution linkage function information indicatinga linkage function to be executed when the deformation operation isperformed, among a linkage function group executable using a combinationof the plural file formats are associated with each other. A linkagefunction associated with a combination of plural file formats isspecified by referring to the linkage function management table.

In Example 12, the linkage function is defined for each file format(attribute). The file format is, for example, the above-describeddocument format, image format, video format, table format, sheet format,graphic format, or audio format.

The linkage function may be a function implemented by linking pluraldifferent functions (e.g., different pieces of software) to each otheror a function implemented by linking identical functions (e.g.,identical software) to each other. For example, when a document creationfunction (document creation software) and an image display function(image display software (e.g., an image viewer)) are linked to eachother, a function of inserting (attaching) an image into a document or afunction of superimposing a document (a character string or a text) onan image is implemented as a linkage function. That is, an imageinsertion function or a text superimposition function is implemented bylinking the document creation function and the image display function toeach other.

The linkage function managed by the linkage function management tablemay be a function executable by software installed in the terminaldevice 10, a function executable by software installed in an externaldevice such as the server 16, or a function executable by softwareinstalled in the terminal device 10 and software installed in anexternal device.

As another example of the linkage function management information,information indicating a combination of plural functions (plural piecesof software), information indicating a linkage function executable bylinking the plural functions (linkage function information), thedeformation operation information, and the execution linkage functioninformation may be associated with each other.

Hereinafter, each linkage function illustrated in FIG. 33 will bedescribed.

In FIG. 33, the “document format” is a text format (with an extension of“.txt”) or a document format (with an extension of “.doc” or “.docx”).The “table format” is a format with an extension expressed as “.csv,”“.xls” or the like. The “image format” is a format with an extensionexpressed as “.jpeg,” “.pdf” or the like. The “video format” is a formatwith an extension expressed as, for example, “.avi,” “.mpeg” or thelike. The “sheet format” is a format with an extension expressed as“.ppt” or the like. While FIG. 33 does not illustrate the graphic formatand the audio format, association between a combination including thegraphic format and the audio format and a linkage function may beregistered in the linkage function management table. Of course, anassociation between a combination including file formats other than thefile formats illustrated in FIG. 33 and a linkage function may bedefined and registered in the linkage function management table.

Each linkage function will be described in more detail. When a documentfile A and a spreadsheet file B are identified as files to be linked(control targets), a “function of inserting a table into a document” anda “function of inserting a document into a table” which are associatedwith a combination of the document format and the table format arespecified as linkage functions. The “function of inserting a table intoa document” is a function of inserting the table represented by thespreadsheet file B into the document represented by the document file A.The “function of inserting a document into a table” is a function ofinserting the document represented by the document file A into the tablerepresented by the spreadsheet file B.

When a document file A and an image file C are identified as files to belinked, a “function of inserting an image into a document” and a“function of superimposing a document on an image” which are associatedwith a combination of the document format and the image format arespecified as linkage functions. The “function of inserting an image intoa document” is a function of inserting the image represented by theimage file C into the document represented by the document file A. The“function of superimposing a document on an image” is a function ofsuperimposing the document represented by the document file A on theimage represented by the image file C.

When a document file A and a video file D are identified as files to belinked, a “function of inserting a video into a document” and a“function of inserting a still image into a document” which areassociated with a combination of the document format and the videoformat are specified as linkage functions. The “function of inserting avideo into a document” is a function of inserting the video representedby the video file D into the document represented by the document fileA. The “function of inserting a still image into a document” is afunction of inserting one or plural frames (still images) constitutingthe video file D into the document represented by the document file A.For example, when the document file A is a document file which is notcompatible with a video, that is, a file into which a video cannot beinserted and when the “function of inserting a still image into adocument” is executed, one or plural frames (still images) included inthe video file D are inserted into the document represented by thedocument file A. Further, a “function of inserting a document into avideo” may be used as a linkage function.

Plural files having the same file format may be identified as files tobe linked. For example, when document files A1 and A2 are identified asfiles to be linked, a “function of integrating documents” which isassociated with the combination of the document format and the documentformat is specified as a linkage function. The “function of integratingdocuments” is a function of integrating the documents represented by thedocument files A1 and A2 with each other (e.g., combining the twodocuments with each other) to generate a new document file A3 whilesaving or removing the document files A1 and A2.

When spreadsheet files B1 and B2 are identified as files to be linked, a“function of integrating tables” which is associated with thecombination of the table format and the table format is specified as alinkage function. The “function of integrating tables” is a function ofintegrating the tables represented by the spreadsheet files B1 and B2with each other (e.g., combining the two tables with each other) togenerate a new spreadsheet file B3 while saving or removing thespreadsheet files B1 and B2.

When video files D1 and D2 are identified as files to be linked, a“function of integrating videos” which is associated with thecombination of the video format and the video format is specified as alinkage function. The “function of integrating videos” is a function ofintegrating the videos represented by the video files D1 and D2 witheach other (e.g., combining the two videos with each other) to generatea new video file D3 while saving or removing the video files D1 and D2.

In addition, when a presentation file E and the document file A areidentified as files to be linked, a “function of inserting a documentinto a presentation sheet” which is associated with the combination ofthe sheet format and the document format is specified as a linkagefunction. The “function of inserting a document into a presentationsheet” is a function of inserting the document represented by thedocument file A into the presentation sheet represented by thepresentation file E.

Hereinafter, Example 12 will be described in detail with reference toFIG. 34. FIG. 34 illustrates the terminal device 10. File images 96 and100 are displayed on the front side display 40 of the terminal device10. The file image 100 is, for example, an image associated with theimage file C. The user may operate the UI unit 20 to change the displaypositions and display sizes of the file images 96 and 100. Further,information indicating file names (for example, character strings) maybe displayed along with the file images 96 and 100 or in place of thefile images 96 and 100.

For example, it is assumed that the user bends the whole of the leftside portion 10 a of the flexible display (terminal device 10 itself)toward the right side portion 10 b of the flexible display. That is, itis assumed that the user bends the flexible display in the direction ofan arrow 102.

The detection unit 24 detects the change (deformation) of the flexibledisplay caused by the above-described bending operation, and specifiesthe position where the change occurs, and the controller 28 specifiesthe file image displayed at that position. When changes occur at thepositions where the file images 96 and 100 are displayed due to theabove-described bending operation, the specifying unit 30 identifies thedocument file A associated with the file image 96 and the image file Cassociated with the file image 100 as control targets (objects to belinked). More specifically, the specifying unit 30 identifies the fileformat of the document file A based on the extension of the documentfile A (e.g., the extension associated with the document file A as theattribute information of the document file A). Similarly, the specifyingunit 30 identifies the file format of the image file C based on theextension of the image file C (e.g., the extension associated with theimage file C as the attribute information of the image file C). Theextension may be included in the file name. When the extension of thedocument file A is the extension representing a document format such as“.txt” or “.docx,” the specifying unit 30 identifies that the fileformat of the document file A is a document format. When the extensionof the image file C is the extension representing an image format suchas “.jpeg,” the specifying unit 30 identifies that the file format ofthe image file A is an image format. Then, by referring to the linkagefunction management table illustrated in FIG. 33, the specifying unit 30specifies the execution linkage function associated with the deformationoperation among the linkage function group associated with thecombination of the document format and the image format. For example, itis assumed that the execution linkage function is a function of“inserting an image into a document.”

For example, the controller 28 causes the flexible display to displayinformation indicating the execution linkage function specified asdescribed above. The controller 28 may output information indicating theexecution linkage function as voice information. When the user gives aninstruction to execute the execution linkage function, the controller 28inserts the image represented by the image file C into the documentrepresented by the document file A.

More specifically, the controller 28 activates a document applicationand an image application. The document application is software (program)having a function of generating, editing, and displaying a file having adocument format. The image application is software (program) having afunction of generating, editing, and displaying a file having an imageformat. The document application may be associated with the documentfile A in advance, and the image application may be associated with theimage file C in advance. The controller 28 uses the functions that theimage application has to copy an image from the image file C and usesthe function that the document application has to paste the image in thedocument represented by the document file A. As a result, a documentinto which an image is inserted is generated. The controller 28 may usethe functions that the image application has to extract a portion imagerepresenting a specific portion from the image represented by the imagefile C and paste the portion image into the document. An area to whichthe image is pasted in the document may be a predetermined area, an areadesignated by the user, or an area presumed that a character stringrelated to the image is described. For example, the controller 28presumes or searches a character string expressing a feature portionrepresented in the image, and presumes, in the document, that an area inwhich a character string having the same or similar meaning as such acharacter string is described is an area in which a character stringrelated to the image is described. Of course, the area may be presumedby another method.

Further, the controller 28 may change the file images 96 and 100.

The controller 28 may generate a new file by executing the linkagefunction. In this case, plural files themselves identified as objects tobe linked are not edited. For example, when the document file A and theimage file C are identified as files to be linked and an executioninstruction of the “function of inserting an image into a document” isgiven, the controller 28 generates a new document file AC by pasting theimage represented by the image file C into the document represented bythe document file A. In this case, the controller 28 generates a newfile image (e.g., an icon) associated with the document file AC, andcauses the flexible display to display the file image. In this case, thedocument file A and the image file C themselves are not edited.

As another example, the controller 28 may edit (including processing orthe like) the files themselves identified as objects to be linkedwithout generating a new file by executing the linkage function. In theabove example, the controller 28 pastes the image represented by theimage file C into the document represented by the document file A, andsaves the document file in which pasting of the image as the documentfile A is reflected. In this case, the document file A is updated.

According to Example 12, when the user deforms the flexible display, thelinkage function according to the deformation is notified or executed.Thereby, the linkage function is notified or executed by a simpleoperation.

Further, similarly to Example 3, the specifying unit 30 may identify thefiles associated with the respective file images displayed on theextension line in the direction of the deformation operation as objectsto be linked (control targets), and specify an execution linkagefunction associated with the deformation operation among the linkagefunction group associated with the combination of the formats of theplural files. The controller 28 notifies or executes the executionlinkage function. The controller 28 may change the linkage functionaccording to the mode of the deformation operation.

Further, Example 4 may be applied to Example 12. That is, the linkagefunction may be changed according to the arrangement order of the pluralfile images with respect to the operation direction.

In Example 12, notification of the information on the linkage functionand the linkage function to be executed may be changed according to amaster-servant relationship between the files. For example, a file whosecontent itself represented by the file is editable is a master file, anda file whose content itself is not editable is a servant file. As anexample, when the document itself represented in the document file iseditable, the document file is the master file. Further, when it isimpossible to edit the image itself represented in the image file (thatis, when it is possible to superimpose content on the image but it isimpossible to edit the image itself), the image file is the servantfile. The same applies to other files. Of course, even for documentfiles, a document file whose document is not editable is a servant file.Even for image files, an image file whose image is editable is a masterfile.

The specifying unit 30 may determine whether the file is either themaster file or the servant file based on the format (i.e., theextension) of the identified file. For example, the specifying unit 30determines that a file having a document format, a file having a tableformat, a file having a sheet format, and a file having a graphic formatare master files, and that a file having an image format, a file havinga video format, and a file having am audio format are the servant files.Of course, the determination as to whether the file is either a masterfile or a servant file may not be made unequivocally based on the fileformat, but the determination as to whether the file is either a masterfile or a servant file may be made by individually determining the fileidentified as the object to be linked. For example, even when a file hasan image format, the file may be a master file so long as the image iseditable.

For example, when the document file A and the image file C areidentified as objects to be linked, the specifying unit 30 determineswhether the document file A is either a master file or a servant filebased on the extension associated with the document file A, anddetermines whether the image file C is either a master file or a servantfile based on the extension associated with the image file C. It isassumed that the extension of the document file A is, for example,“.docx,” and that the extension of the image file C is, for example,“.jpeg.” In this case, since the document file A has the document formatand is a file that is editable, the specifying unit 30 determines thatthe document file A is the master file. Further, since the image file Chas the image format and is a file that is not editable, the specifyingunit 30 determines that the image file C is the servant file. Of course,the specifying unit 30 may determine whether the document file A and theimage file C are editable files, by determining the document file A andthe image file C individually without making a determination based onthe extension.

Since the document file A is the master file and the image file C is theservant file, the image represented in the image file C is the contentto be pasted, and the document represented by the document file A is thecontent to which the image is pasted. In this case, the specifying unit30 specifies the “function of inserting an image into a document” as anexecution linkage function.

Further, among the plural linkage functions registered in the linkagefunction management table illustrated in FIG. 33, the controller 28 maynotify the information on the linkage function, which is specified basedon the master-servant relationship of the files, in preference to theinformation on the other linkage functions. Descriptions will be made onthe document file A and the image file C by way of an example. As alinkage function associated with a combination of a document format andan image format, a “function of inserting an image into a document” anda “function of superimposing a document on an image” are registered inthe linkage function management table (see FIG. 33). In this case, thespecifying unit 30 specifies the “function of inserting an image into adocument” which is specified based on the master-servant relationship ofthe files as a linkage function having “1st” priority and specifies the“the function of superimposing a document on an image” as a linkagefunction having “2nd” priority. According to the priorities, thecontroller 28 notifies the information on the “function of inserting animage into a document” in preference to (e.g., at a higher rank than)the information on the “function of superimposing a document on animage.”

Further, when plural linkage functions are specified based on themaster-servant relationship of the files, the controller 28 maydetermine the priority of each linkage function based on the use historyof each linkage function by the user, and notify the information on eachlinkage function according to the priority of each linkage function. Forexample, the controller 28 notifies the information on the linkagefunction with high frequency of use preferentially (e.g., at a higherrank).

Further, when plural linkage functions are specified based on themaster-servant relationship of the files, the controller 28 maydetermine the priorities of the linkage functions based on therelationship between plural files identified as objects to be linked(i.e., the relationship between plural file formats), and notify theinformation on the respective linkage functions according to thepriorities of the linkage functions. For example, the controller 28determines the possibility that each linkage function included in theplural linkage functions specified by the specifying unit 30, based onthe plural file formats. For example, the controller 28 notifies theinformation on the linkage function with a higher use possibilitypreferentially (e.g., at a higher rank). For example, in the combinationof the document format and the image format, it is expected that the“function of inserting an image into a document” is more likely to beused than the “function of superimposing a document on an image.” Inthis case, the controller 28 notifies the information on the “functionof inserting an image into a document” in preference to the informationon the “function of superimposing a document on an image.”

The combinations of objects to be linked is not limited to thecombinations in the first to fourth exemplary embodiments. For example,when a combination of at least two of devices, functions (includingfunctions executed by devices and functions executed by software),files, or software is identified as objects to be linked, notificationor execution of a linkage function executable using the combination maybe controlled. For example, when a device and a file are identified asobjects to be linked as a result of the deformation operation on theflexible display, the specifying unit 30 specifies a linkage functionexecutable using the device and the file, and the controller 28 notifiesor executes the linkage function. As a specific example, when amultifunction device and a document file are identified as objects to belinked, the specifying unit 30 specifies linkage functions executableusing the multifunction device and the document file (e.g., a functionof printing the document file by the multifunction device, a function offacsimile transmission of the document file by the multifunction device,etc.). The controller 28 notifies or executes the linkage functions.Similarly, when a function and a file are identified as objects to belinked, the specifying unit 30 specifies a linkage function executableusing the function and the file, and the controller 28 notifies orexecutes the linkage function. As a specific example, when a documentapplication and a document file are identified as objects to be linked,the specifying unit 30 specifies a linkage function executable using thedocument application and the document file (e.g., a function of editingor outputting the document file by the document application, etc.).Further, when a device, a function, and a file are identified as objectsto be linked, the specifying unit 30 specifies a linkage functionexecutable using the device, the function (e.g., software), and thefile, and the controller 28 notifies or executes the linkage function.As a specific example, when a multifunction device, a documentapplication, and a document file are identified as objects to be linked,the specifying unit 30 specifies a linkage function executable using themultifunction device, the document application, and the document file(e.g., a function of editing the document file by a documentapplication, outputting the document to the multifunction device, andprinting and facsimile transmission of the document file by themultifunction device, etc.). The combinations in the above-describedspecific examples are merely illustrative, and linkage functionsexecutable by combining devices, functions, files, and software otherthan those described above may be defined.

Further, in the first to fourth exemplary embodiments, when the samedeformation operation is successively performed on the flexible display(e.g., when the same deformation operation is performed twice within apredetermined time), the controller 28 may cancel the notification orexecution of a solo function or a linkage function specified by thefirst deformation operation. That is, when the same deformationoperation is performed successively, notification or execution of thesolo function or the linkage function is canceled. In this manner, it ispossible to cancel the notification or execution of the functions by asimple operation.

Further, the detection unit 24 may detect the deformation operation on aspecific portion of the flexible display as an invalid operation. Forexample, a portion assumed to be gripped by the user to hold theflexible display is the specific portion. By providing such a specificportion, when the user grips the specific portion to hold the flexibledisplay, the gripped portion is not detected as a portion subjected tothe deformation operation, so that the operation not intended by theuser is not performed. The specific portion may be determined in advanceor may be set by the user.

Further, the specifying unit 30 may identify a control target accordingto a deformation operation, and the controller 28 may cause the displayunit of the UI unit 20 to display a function selection screen allowing auser to select a solo function or a linkage function executable usingthe control target. When the user selects the solo function or thelinkage function on the function selection screen and gives an executioninstruction, the controller 28 controls the execution of the functionselected by the user.

Further, when a deformation operation is performed on the flexibledisplay for a predetermined time or longer, the detection unit 24 maydetect the deformation operation as a valid operation. In addition, thedetection unit 24 may detect a deformation operation performed for atime less than the predetermined time as an invalid operation. Thus, anaction not intended by the user is not performed. The above-mentionedpredetermined time may be changed by the user.

Further, the detection unit 24 may determine whether the deformationoperation is valid or invalid based on the magnitude of a force appliedto the flexible display per unit time, or may determine whether thedeformation operation is valid or invalid based on a pressure applied tothe flexible display.

Fifth Exemplary Embodiment

Hereinafter, a flexible display according to a fifth exemplaryembodiment will be described. In the fifth exemplary embodiment, thedetection unit 24 detects a change in the flexible display from areference position, and the controller 28 controls a process accordingto the change amount and changes the reference position. Hereinafter,the fifth exemplary embodiment will be described in detail.

FIG. 35 schematically illustrates a state where the flexible display(the terminal device 10 itself in the example illustrated in FIG. 35) isseen from the lateral side. A reference position 104 is a position whenthe flexible display is not bent (default position). When the user bendsa portion 106 of the flexible display to a position of a broken lineportion 108, the detection unit 24 detects a bending amount. Forexample, the detection unit 24 detects an angle θa between the referenceposition 104 and the position of the broken line portion 108 as abending amount. The controller 28 controls notification and execution ofa process according to the bending amount.

For example, the bending amount and a function are associated with eachother, and information indicating the association is stored in theterminal device 10, the server 16, or the like. The specifying unit 30specifies a function associated with the bending amount (e.g., the angleθa) detected by the detection unit 24, and the controller 28 controlsnotification and execution of the specified function. The function maybe a function executed by a device, a function executed by software, asolo function executed by a single control target (a device orsoftware), or a linkage function executed by one or plural controltargets.

Further, when the bending amount is detected by the detection unit 24while application software is activated, the controller 28 may cause theapplication software to execute a process according to the bendingamount. For example, with respect to the same application software, aprocess executed by the application software is associated with eachbending amount, and the specifying unit 30 specifies a processassociated with the bending amount detected by the detection unit 24.The controller 28 controls notification and execution of the specifiedprocess.

For example, the controller 28 changes the reference position accordingto a predetermined change condition. The change condition is, forexample, a condition related to a timing of changing the referenceposition. Specifically, the controller 28 changes the reference positionbased on the change amount at a time at which the flexible displaystarts to display. The time at which it is started to display is, forexample, a timing at which the power supply of the flexible display isturned on, a timing at which the flexible display resumes from the sleepstate or the pause state to the operating state (i.e., a timing at whichrestriction on the power consumption is canceled), or the like. Further,the controller 28 may change the reference position when the flexibledisplay does not change for a predetermined time or more.

For example, the detection unit 24 detects a bending position of theflexible display when the power supply of the flexible display is turnedon, and the controller 28 uses the detected position as a referenceposition. When the bending position is not detected, the controller 28uses the default reference position 104. Meanwhile, when the portion 106is bent to the position of the broken line portion 108 and the bendingposition (the position of the broken line portion 108) is detected whenthe power supply is turned on, the controller 28 uses the position ofthe broken line portion 108 (the position of the angle θa) as a newreference position. The same is true when the flexible display resumesfrom the sleep state or the pause state. That is, the detection unit 24detects the bending position of the flexible display when the flexibledisplay resumes from the sleep state or the pause state to the operatingstate, and the controller 28 uses the detected position as a referenceposition. When the bending position is not detected, the controller 28uses the default reference position 104.

Further, when the portion 106 is bent to the position of the broken lineportion 108 and the bending state is kept for a predetermined time ormore and detected by the detection unit 24, the controller 28 may usethe position of the broken line portion 108 as a new reference position.

When a new reference position is determined, the controller 28 controlsnotification and execution of a process according to the bending amountfrom the new reference position.

Hereinafter, a process of changing the reference position will bedescribed in detail with reference to FIG. 36. FIG. 36 schematicallyillustrates a state where the flexible display (the terminal device 10itself in the example illustrated in FIG. 36) is seen from the lateralside. For example, the portion 106 of the flexible display is bent to aposition of a solid line portion 110. The angle between the originalreference position 104 and the solid line portion 110 is an angle θx.When the change condition is satisfied, the controller 28 changes thereference position from the original reference position 104 to theposition of the solid line portion 110. That is, the controller 28 usesthe position of the solid line portion 110 as a new reference position.When the portion 106 is bent from the new reference position (theposition of the solid line portion 110) to a position of a broken lineportion 112 (a position at which an angle with the new referenceposition is an angle θa), the specifying unit 30 specifies a processassociated with the bending amount (angle θa), and the controller 28controls notification or execution of the process. Therefore, theprocess according to the bending amount is appropriately executed. Inthis regard, detailed descriptions will be made with reference to FIGS.36 and 37.

FIG. 37 schematically illustrates a state where the flexible display(the terminal device 10 itself in the example illustrated in FIG. 37) isseen from the lateral side. Consideration will be made on, for example,a case where the portion 106 is bent to the position of the solid lineportion 110 and its state is kept, and the portion 106 is bent from sucha position to a broken line portion 108. A position of the broken lineportion 108 is a position where the portion 106 is bent from theoriginal reference position 104 by the angle θa. When the originalreference position 104 is used as the reference position and the portion106 is bent from the solid line portion 110 to the broken line portion108, the detection unit 24 detects the angle θa as the bending amount.Although the angle between the solid line portion 110 and the brokenline portion 108 is an angle θy (<θa) and the actual bending amount isthe angle θy, the angle θa is detected as the bending amount, so that aprocess associated with the angle θa is notified or executed. To thecontrary, in the case where the solid line portion 110 is used as thereference position, even when the portion 106 is bent from the solidline portion 110 to the broken line portion 108, the bending amount(angle θy) is detected and the bending amount (angle θa) is notdetected. When the portion 106 is bent to the broken line portion 112 asillustrated in FIG. 36, the bending amount (angle θa) is detected and aprocess associated with the angle θa is notified or executed. Thereby,it is possible to prevent the occurrence of the operation not intendedby the user.

Hereinafter, the operation according to the fifth exemplary embodimentwill be described. First, the controller 28 detects the current state ofthe flexible display (e.g., the state of the power supply (ON, OFF,sleep, pause, etc.), application software during activation, etc.).Next, the detection unit 24 detects the current bending position of theflexible display. For example, when the portion 106 is bent to theposition of the solid line portion 110 as illustrated in FIG. 36, thedetection unit 24 detects such a bending position. In a case where theflexible display is a slide type display, the detection unit 24 maydetect the position of the display when the flexible display isdeveloped in a sliding manner. Next, the controller 28 sets a referenceposition of the deformation operation. When the flexible display is notbent as illustrated in FIG. 35, the controller 28 sets the defaultreference position 104 as a reference position. When the portion 106 isbent to the position of the solid line portion 110 as illustrated inFIG. 36, the controller 28 sets the position of the solid line portion110 as a reference position. Further, when the application software isactivated, the controller 28 identifies the activated applicationsoftware. Next, when a deformation operation (e.g., a bending operation)is performed on the flexible display, the detection unit 24 detects thebending amount from the reference position set as described above. Thespecifying unit 30 specifies a process according to the bending amount,and the controller 28 notifies or executes the specified process.

According to the fifth exemplary embodiment, it is possible to perform acontrol based on a change from the reference position to which thechange of the flexible display as a display is reflected. Thereby, it ispossible to prevent, for example, the occurrence of the operation notintended by the user. For example, in a case where the flexible displayhas a bending trace, a case where the flexible display may be kept in abent state by using a gear or the like, or a case where a materialcapable of keeping the flexible display in the bent state is used, whena process according to the bending amount from the default referenceposition is notified or executed, the process unintended by the user maybe notified or executed. According to the fifth exemplary embodiment,even though the flexible display is in such a state, a process accordingto the bending amount from a new reference position is notified orexecuted. Thus, it is possible to prevent the occurrence of the aboveproblem.

For example, it is assumed that specific application software isexecuted when a portion of the flexible display is bent by 60 degrees.In this case, when a portion of the flexible display is already bent by30 degrees from the default reference position, the specific applicationsoftware will be executed merely by bending the portion by further 30degrees. Since the user intends that the specific application softwareis executed when the flexible display is bent by 60 degrees, the aboveoperation is contrary to the user's intention. To the contrary,according to the fifth exemplary embodiment, the position where theflexible display is bent by 30 degrees from the default referenceposition is defined as a new reference position. Therefore, even whenthe flexible display is further bent by 30 degrees from the newreference position, the specific application is not executed. Thespecific application software is executed when the flexible display isfurther bent by 60 degrees from the new reference position. Therefore,an operation consistent with the intention of the user is implemented.

Modification 1

Hereinafter, Modification 1 of the fifth exemplary embodiment will bedescribed with reference to FIG. 37. In Modification 1, a user selectseither a reference position before change (e.g., the reference position104) or a reference position after change (e.g., the position of thesolid line portion 110), and the controller 28 controls a processaccording to the change amount from the reference position selected bythe user. Information indicating the reference position before change(past reference position) is stored in the terminal device 10.

For example, as illustrated in FIG. 37, it is assumed that the portion106 is already bent to the position of the solid line portion 110. Whenthe reference position after change is selected, as in the aboveexample, the controller 28 uses the position of the solid line portion110 as a reference position, the specifying unit 30 specifies a processaccording to the bending amount from the position of the solid lineportion 110, and the controller 28 notifies or executes the specifiedprocess. For example, when the portion 106 is bent to the position ofthe broken line portion 112 (the position of the angle θa) asillustrated in FIG. 36, the controller 28 notifies or executes theprocess associated with the angle θa.

Meanwhile, when the reference position before change is selected, thecontroller 28 uses the default reference position 104 as a referenceposition, the specifying unit 30 specifies a process according to thebending amount from the reference position 104, and the controller 28notifies or executes the specified process. For example, when theportion 106 is bent from the position of the solid line portion 110 tothe position of the broken line portion 108 (the position of the angleθy with respect to the position of the solid line portion 110 as areference and the position of the angle θa with respect to the referenceposition 104 as a reference) as illustrated in FIG. 37, the controller28 notifies or executes a process according to the bending amount (angleθa) from the reference position 104 to the position of the broken lineportion 108.

According to Modification 1, when the reference position before changeis selected, the process may be notified or executed with less bendingamount as compared with the case where the reference position afterchange is selected.

Modification 2

Hereinafter, Modification 2 of the fifth exemplary embodiment will bedescribed with reference to FIG. 38. FIG. 38 is a schematic diagramillustrating a state where the flexible display (the terminal device 10itself in the example illustrated in FIG. 38) is seen from the lateralside. In Modification 2, the change amount is detected stepwise, and thecontroller 28 controls a process according to the detected changeamount. Hereinafter, Modification 2 will be described in detail withreference to FIG. 38.

For example, when the flexible display is bent from the defaultreference position 104 to the position of a broken line portion 114 (aposition of an angle θb), the specifying unit 30 specifies a function Aassociated with the bending amount (a bending amount corresponding tothe angle θb), and the controller 28 notifies or executes a processusing the function A. For example, when the flexible display is furtherbent from the position of the broken line portion 114 to a position of abroken line portion 116 (the position of the angle θb from the positionof the broken line portion 114, that is, the position of the angle 20 bfrom the reference position 104) within a predetermined time limit fromthe time when the flexible display is bent to the position of the brokenline portion 114, the specifying unit 30 specifies a function Bassociated with the bending amount (the bending amount corresponding tothe angle 20 b), and the controller 28 notifies or executes a processusing the function B. After the above-mentioned time limit elapses, thecontroller 28 uses the position of the broken line portion 114 as areference position. In this case, when the flexible display is furtherbent from the position of the broken line portion 114 to the position ofthe broken line portion 116, the specifying unit 30 specifies thefunction A associated with the bending amount (the bending amountcorresponding to the angle θb), and the controller 28 notifies orexecutes the process using the function A.

As another example, it is assumed that the flexible display is bent tothe position of the broken line portion 114 (the position of the angleθb), and the bending amount (the bending amount corresponding to theangle θb) is detected by the detection unit 24. Next, it is assumed thatthe flexible display is further bent from the position of the brokenline portion 114 to the position of the broken line portion 116 (abending operation corresponding to the angle θb is further performed)within the predetermined time limit (time for establishing thedeformation operation) from the time when the bending amount isdetected, and the second bending operation is detected by the detectionunit 24. In this case, the specifying unit 30 may specify a function Cassociated with the two-step bending operation, and the controller 28may notify or execute a process using the specified function C. That is,when the flexible display is bent from the reference position 104 to theposition of the broken line portion 114 (the angle θb), the bendingamount (the bending amount corresponding to the angle θb) is detected bythe detection unit 24, and the time limit elapses in that state, thedeformation operation is established, and the controller 28 notifies orexecutes the process using the function A. Further, when the flexibledisplay is bent from the reference position 104 to the position of thebroken line portion 116 (the angle 20 b), and the bending amount (thebending amount corresponding to the angle 20 b) is detected by thedetection unit 24 and when the time limit elapses in that state, thedeformation operation is established, and the controller 28 notifies orexecutes the process using the function B. Further, when the flexibledisplay is bent to the position of the broken line portion 114 (theposition of the angle θb), the bending operation is detected by thedetection unit 24, and the time limit elapses, the position of thebroken line portion 114 is determined as the reference position. Then,when the flexible display is further bent from the position of thebroken line portion 114 to the position of the broken line portion 116(the bending operation corresponding to the angle θb is furtherperformed), the bending amount (the bending amount corresponding to theangle θb) is detected by the detection unit 24, and the time limitelapses in that state, the deformation operation is established, and thecontroller 28 notifies or executes the process using the function A.Also, when the flexible display is bent to the position of the brokenline portion 114 (the position of the angle θb), the bending operationis detected by the detection unit 24, then, the flexible display isfurther bent from the position of the broken line portion 114 to theposition of the broken line portion 116 (the bending operationcorresponding to the angle θb is further performed) within a time limit(before the deformation operation is established), the second bendingoperation is detected by the detection unit 24, and the time limitelapses in that state, the deformation operation is established, and thecontroller 28 notifies or executes the process using the function C.

According to Modification 2, the process according to the change amountdetected stepwise is notified or executed, so that the number ofexecutable processes may be increased even within the same range of thechange amount.

Modification 3

Hereinafter, Modification 3 of the fifth exemplary embodiment will bedescribed with reference to FIG. 39. FIG. 39 schematically illustrates astate where the flexible display (the terminal device 10 itself in theexample illustrated in FIG. 39) is seen from the lateral side. InModification 3, the controller 28 controls a process according to achange direction of the flexible display.

The change amount, the change direction, and the function are associatedwith each other. The specifying unit 30 specifies a function associatedwith the change amount and the change direction which are detected bythe detection unit 24, and the controller 28 notifies or executes aprocess using the specified function.

For example, when the portion 106 of the flexible display is bent fromthe reference position 104 to a position of a broken line portion 118 asillustrated in FIG. 39, the detection unit 24 detects the bending amountand the bending direction. In the example illustrated in FIG. 39, anangle +θa is detected as the bending amount and the bending direction.The specifying unit 30 specifies a function associated with the angle+θa, and the controller 28 notifies or executes a process using thespecified function.

Further, when the portion 106 of the flexible display is bent from thereference position 104 to a position of a broken line portion 120, thedetection unit 24 detects its bending amount and its bending direction.The position of the broken line portion 120 is a position opposite tothe broken line portion 118 across the reference position 104, and anangle −θa is detected as the bending amount and the bending direction.The specifying unit 30 specifies a function associated with the angle−θa, and the controller 28 notifies or executes a process using thespecified function. The angle −θa is associated with a functiondifferent from the function associated with the angle +θa. Thus, whenthe flexible display is bent to the position of the angle −θa, a processusing the function different from the case where the flexible display isbent to the position of the angle +θa, is notified or executed.

According to Modification 3, since the process according to the changeamount and the change direction is controlled, more processes may becontrolled, as compared with the case where the process according to thechange amount only or the change direction only is controlled.

Each of the terminal device 10, the devices 12 and 14, and the server 16is implemented by, for example, cooperation of hardware and software.Specifically, each of the terminal device 10, the devices 12 and 14, andthe server 16 includes one or plural processors such as CPUs (notillustrated). As the one or plural processors read and execute a programstored in the storage device (not illustrated), the functions of eachpart of the terminal device 10, the devices 12 and 14, and the server 16are implemented. The program is stored in a storage device via arecording medium such as a CD or a DVD, or via a communication path suchas a network. As another example, each part of the terminal device 10,the devices 12 and 14, and the server 16 may be implemented by hardwareresources such as a processor, an electronic circuit, an applicationspecific integrated circuit (ASIC), or the like. A device such as amemory may be used in the implementation. As still another example, eachunit of the terminal device 10, the devices 12 and 14, and the server 16may be implemented by a digital signal processor (DSP), a fieldprogrammable gate array (FPGA), or the like.

The foregoing description of the exemplary embodiments of the presentinvention has been provided for the purposes of illustration anddescription. It is not intended to be exhaustive or to limit theinvention to the precise forms disclosed. Obviously, many modificationsand variations will be apparent to practitioners skilled in the art. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, therebyenabling others skilled in the art to understand the invention forvarious embodiments and with the various modifications as are suited tothe particular use contemplated. It is intended that the scope of theinvention be defined by the following claims and their equivalents.

What is claimed is:
 1. An information processing device comprising: acontroller that controls an instruction to a control target associatedwith an image displayed at a changed position of a deformable display,according to a detection result of a change in each position on thedeformable display, wherein the controller controls notification of afunction executable using the control target associated with the imagedisplayed at the changed position, and when a specific deformation ofthe deformable display is detected at the changed position, thecontroller controls notification of a linkage function executable usinga plurality of the control targets associated with the image displayedat the changed position, wherein the controller changes the linkagefunction according to a mode of change of the deformable display, thechange of the deformable display is a bending change of the deformabledisplay, and the controller changes the linkage function according to amode of the bending change.
 2. The information processing deviceaccording to claim 1, wherein the controller further changes the imagedisplayed at the changed position, according to the detection result. 3.The information processing device according to claim 2, wherein when thecontrol target associated with the image displayed at the changedposition is not controllable, the controller returns the image after thechange to an original image and displays the original image.
 4. Theinformation processing device according to claim 1, wherein when aplurality of the images associated with the plurality of control targetsare displayed at a position corresponding to the specific deformation,the controller controls notification of the linkage function executableusing the plurality of control targets.
 5. The information processingdevice according to claim 1, wherein the change of the deformabledisplay is a bending change of the deformable display, and thecontroller controls notification of a function executable using acontrol target associated with an image displayed on an extension lineof a bending direction.
 6. The information processing device accordingto claim 5, wherein when one image associated with one control target isdisplayed on the extension line, the controller controls notification ofa solo function executable using the one control target.
 7. Theinformation processing device according to claim 5, wherein when aplurality of images associated with a plurality of control targets aredisplayed on the extension line, the controller controls notification ofa linkage function executable using the plurality of control targets. 8.The information processing device according to claim 7, wherein thecontroller changes the linkage function according to an arrangementorder of the plurality of images on the extension line.
 9. Theinformation processing device according to claim 5, wherein thecontroller controls notification of a function executable using aportion of the control target associated with the image displayed on theextension line.
 10. The information processing device according to claim1, wherein the deformable display includes a first screen that displaysa state of the control target viewed from a first direction, and asecond screen that displays a state of the control target viewed from asecond direction, and the controller performs a control according to therespective changes in the first screen and the change in the secondscreen.
 11. The information processing device according to claim 10,wherein when a valid change with respect to the first screen isdetected, the controller controls an instruction to a functionassociated with the first direction side of the control target, and whena valid change with respect to the second screen is detected, thecontroller controls an instruction to a function associated with thesecond direction side of the control target.
 12. The informationprocessing device according to claim 11, wherein a bending operation ofthe deformable display toward the first screen side is detected as avalid change with respect to the first screen, and a bending operationof the deformable display toward the second screen is detected as avalid operation with respect to the second screen.
 13. The informationprocessing device according to claim 1, wherein the control target is atleast one of a device, a function, a file, or software.
 14. Theinformation processing device according to claim 1, further comprising:the deformable display.
 15. An information processing method comprising:controlling an instruction to a control target associated with an imagedisplayed at a changed position of a deformable display, according to adetection result of a change in each position on the deformable display;controlling notification of a function executable using the controltarget associated with the image displayed at the changed position;controlling notification of a linkage function executable using aplurality of the control targets associated with the image displayed atthe changed position in response to determining that a specificdeformation of the deformable display is detected at the changedposition; and changing the linkage function according to a mode ofchange of the deformable display, wherein the change of the deformabledisplay is a bending change of the deformable display, and changing thelinkage function according to a mode of change of the deformable displayincludes changing the linkage function according to a mode of thebending change.
 16. A non-transitory computer readable medium storing aprogram causing a computer to execute information processing, theinformation processing comprising: controlling an instruction to acontrol target associated with an image displayed at a changed positionof a deformable display, according to a detection result of a change ineach position on the deformable display; controlling notification of afunction executable using the control target associated with the imagedisplayed at the changed position; and controlling notification of alinkage function executable using a plurality of the control targetsassociated with the image displayed at the changed position in responseto determining that a specific deformation of the deformable display isdetected at the changed position; and changing the linkage functionaccording to a mode of change of the deformable display, wherein thechange of the deformable display is a bending change of the deformabledisplay, and changing the linkage function according to a mode of changeof the deformable display includes changing the linkage functionaccording to a mode of the bending change.